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ABSTRACT 

A group of seven conference papers, all dealing with 
the role of testing and evaluation in the schools, is presented in 
this document* The papers are as follows: (1) "Framework for 
Effective Schools" (Eric Cooper), which identifies indicators of 
efficacy, quality, and equity; (2) "Quality Indicators for Monitoring 
Equity" (Ramsay Selden), which calls for a reform of data collection 
on how schools are functioning for underachieving minorities; (3) 
"Persistence and Patience" (Todd Endo), which discusses the 
development and evaluation of the Fairfax County (Virginia) Public 
Schools* minority students' achievement plan; (4) "Monitoring and 
Improving School Learning* 9 (Halter Hathaway), which discusses 
improvements in achievement testing and test results made by the 
Portland, Oregon, school district; (5) "The Need to Assess Multiple 
Crucial Components in Evaluating Programs" (Daniel Levine), which 
illustrates the importance of attending to multiple components of 
effective programs and determining how they may improve instruction 
for disadvantaged minority students; (6) "Some Thoughts on How 
Testing and Evaluation Can Improve Educational Opportunities for 
Underachieving Minorities" (Ernesto M* Bernal), which suggests that 
evaluation and testing can improve programs for underachieving 
minority students by documenting progress; and (7) "Results from 
Using the HI CAT Learning Solution for Underachieving Minority 
Students* 9 (James B. Olsen) * Some of the papers contain data tables 
and figures, and/or include lists of references* (BJV) 
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THE ROLE OF TESTING AND EVALUATION 



Framework for Effective Schools 1 

Eric Cooper 
College Board 
Ad Hoc Committee on Effective Schools 

The strength of our nation is a direct function of the ability of 
o r schools to educate — majority as well as minority, women as well 
as men, children as well as adults. All students served — all students 
educated. This is not only a goal, but a necessity. An institution that 
provides "effective schooling" is one that is able to maintain 
sustained progress towards national goals and expectations for all 
students. 

In this definition, it is important to note that a school with a 
"good" reputation may not be an effective school. The difference 
between' the two relates to the concept of progress and whether or 
not all students are well served. To some, "good schools" develop 
high levels of achievement for a certain portion of the students 
served. Effective schools, on the other hand, hold to a higher 
standard. They sustain high levels of growth for all students. 

The recent reports on reading and writing produced by the 
National Assessment of Educational Progress document well the fact 
that the average performance of students on the process outcomes of 
education is simply not high enough to meet the needs of the Nation. 
The Nation is still at risk. Therefore, for effective schooling to 
become a reality, our schools must set and hold all students to higher 
expectations of performance on the process outcomes of education. 

In order to provide effective schooling, u-i institution must set 
high goals and expectations, not only for the students served, but 
also for its administrators and its teachers and parents. And these 
goals and expectations must apply equally to students who are 
considered academically gifted and those who are considered 
educationally at risk. White equality in outcomes cannot be 
guaranteed, there is no place for differential opportunities or 
expectations for students. 

! This framework was developed by the Ad Hoc Committee 
on Effective Schools 
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Stating that an institution has high expectations for the students it 
serves is not enough for effective schooling. These expectations must 
be stated in concrete terms — in ways that can be observed and 
assessed. And as an integral part of effective schooling, student 
progress towards these expectations must be monitored continuously 
- by measures of valued educational outcomes as well as by other 
indicators of success. An example of the former would be the 
periodic assessment of student ability to write through the use of 
direct writing samples. An example of the latter would be evidence 
that the retention rate for at-risk students was as high as the 
retention rate for the academically talented. 

In stating that measures of valued educational outcomes must 
be used to monitor student progress, it is important to define what is 
meant by Valued educational outcomes/ Education is not simply the 
acquisition of subject matter knowledge — the facts and figures. 
Effective schooling involves the processes of learning the processes 
needed to actually solve problems, read with comprehension, and 
develop an idea in writing, for example. And measures of these 
processes, rather than of the discrete skills and facts, must be used to 
assess student progress towards these valued outcomes. No school 
can be considered effective without evidence of sustained progress 
for all students on these processes of learning. 

In order to ensure that instruction is fulfilling student needs, 
appropriate tests that are in accord with real-world outcomes should 
be used. These tests should repeatedly show that students in all 
classrooms and at all ability levels are making satisfactory progress 
toward stated goals and expectations. They should demonstrate that 
students are successful on those optimally difficult learning tasks 
that ensure growth. 

While it is important for teachers to use tests to monitor 
instruction, it is very important to distinguish these tests from those 
which show that students are making progress toward valued 
educational outcomes. These outcome measures should be used for 
the purpose of redesigning the curriculum and teacher-directed 
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instruction so that ever increasing progress is made towards the 
bottom line — towards attaining valued outcomes. 

Two statewide testing programs stand out as exemplars in 
terms of their use of measures that focus on the process outcomes of 
education rather than the discrete skills and facts of education. In 
Connecticut and New York, outcome measures are used that permit 
the assessment of student progress towards expectations — absolute 
expectations set in terms of functional needs of the State and the 
Nation rather than relative needs set in relation to the average 
performance of students on tests that are only norm- referenced. 

Effective schools use a variety of assessments to provide 
information for informed decision-making not only in terms of 
current problems, but also with respect to the need for attaining 
those levels of performance required to function effectively in the 
future. According to NAEP, for example, only 1 in 20 seventeen- 
year-old students can read at the "adept" level. It is obvious that we 
need to know what must be done in the future in order to plan for 
new and higher levels of performance. 

Inadequate student performance is a cause for restructuring 
and redesign not only of educational programs, but also of the 
retooling of management and related support systems in the school 
and in the community which are required for implementation of 
instructional programs that have been shown to work in similar 
educational systems. Effective schools are problem-solving 
institutions which serve all students in meeting their aspirations and 
the nation's need for informed, literate citizens in an increasingly 
competitive world. 

In order to define and resolve problems, and sustain high 
levels of progress for all students, effective schools: 

• Reallocate and/or allocate additional resources (e.g., time, 
teachers, and materials) to improve the performance of low- 
achieving students. 

• Involve teachers and other staff in ongoing examination and 
revision of decisions and in collegial problem-solving regarding 
effective implementation of instruction within and across 
classrooms. 
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Provide adequate resources to improve the delivery of 
instruction of the most important learning skills, particularly for 
low-achieving students, through school-wide staff development. 



In drawing attention to the processes of learning, such as the 
ability to solve problems, read with comprehension, and develop an 
idea in writing, we must take note of the fact that effective schooling 
requires that students be engaged in productive learning 
experiences, not simply time-on-task. And throughout the school 
year, the amount of time students are actively engaged in the 
learning process is critically important. Furthermore, teachers must 
be sensitive to the needs and abilities of the students served in order 
to engage them in productive learning experiences in the classroom. 
This means that the materials used to deliver instruction must not be 
so easy as to create boredom, nor too difficult to create frustration. 

In effective schools, all students must have a demonstrable 
opportunity to develop cognitive processes to comprehend, think, 
and compute. This means that students should be actively engaged 
in a mixture of interactive and teacher-directed instruction for a 
significant portion of the school day. All students should also have 
an opportunity to learn in the content fields and such opportunities 
should be integrated with the development of processing capabilities. 
While the curriculum may be enriched for the gifted, it should not be 
trivialized for those who are at risk or who are more dependent upon 
the school for their development. Finally, effective schooling 
requires that teachers be sensitive to the art of classroom 
questioning, listening well to student responses and providing 
constructive clues and feedback to facilitate the learning process. 

While the above generalities hold for institutions that provide 
effective schooling, there are a number of additional requirements 
for institutions that serve high-risk students — students, who are 
almost solely dependent upon schools for the development of the 
processes of education. The educational attainments of 
disadvantaged students, who are disproportionately black, Hispanic, 
American Indian, and poor serve as a barometer to determine 
whether a school is engaged in effective schooling. An effective 
school produces as much progress towards national goals and 
expectations for these students as it does tor the educationally 
advantaged. 
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To provide effective schooling for the educationally 
disadvantaged, a school must assign some of the best teachers, 
allocate a disproportionate amount of resources, and provide small 
class sizes or otherwise address the problems of these students. 
Anything less, and the educationally disadvantaged will not be able 
to sustain progress towards the acquisition of the processes of 
learning. Finally, to sustain progress, particularly for at-risk 
students, attendance should be high and the school should 
aggressively resist the transferring of students in and out of 
classrooms for pull-out programs, unless they are fully and 
productively coordinated with regular classroom instruction. 

Effective schooling requires that concrete arv' manageable 
plans be developed and in place for starting off a school year with 
complete programs and a fully professional staff ready to teach. The 
school year is too short to waste time on start-up processes and it is 
especially important to make sure that the at-risk students are 
served with the best staff from the very start of the school year. 
Effective schooling requires the systemtic upgrading of instruction so 
that it is in accord with the state-of-the-art in instructional fields as 
documented in various professional reports such as the N1E report 
entitled, Becoming A Nation of Readers. 

Teachers and administrators who provide effective schooling 
are critical consumers of educational books, computer software, and 
other products, making sure that they have been validated for 
instructional use. Because these products overwhelmingly determine 
the nature of instruction, those involved in effective schooling 
frequently acquire updated products to support efforts at upgrading 
themselves. 

Effective schools recognize the importance of teachers, parents, 
and community representatives and involve them in the planning, 
decision-making, and evaluation process of educating children. 
Effective schools do not depend on top-down mandates to improve 
instruction. Effective implementation of instructional reform 
requires a mixture of school-level decision-making and top-level 
direction-setting, with emphasis on on-going, building-level staff 
development and initiative focusing on how instructional 
improvements will be defined, implemented, and modified. Teachers 
must be deeply involved as scholar practitioners in determining, 
through collegial decision making, how improvements are shaped 
and delivered. 
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Evaluation indicators and criteria that can be employed in 
identifying institutions that provide effective schooling can be 
grouped under three major headings: (1) indicators of efficacy, (2) 
indicators of quality, and (3) indicators of equality. Examples of such 
criteria axe as follows: 

(1) fodicators of efficacy: 

• Assessment of educational outcomes based on process 
measures such as work samples, direct writing samples, and 
holistic measures of comprehension. Specifically, tests 
which sample discrete skills rather than engage 
comprehension, writing, and computing processes should not 
be relied upon as indicators of educational progress. 
Attempts to legislate improvements in education through 
minimum competency testing programs fall short for 
students, especially students who are at risk, because they 
focus attention on lower-level discrete skills at the expense 
of comprehension, problem-solving, and the expression of 
ideas orally and in writing. 

• Frequent monitoring of student progress towards outcomes 
by classroom teachers using a variety of formal and 
informal procedures. 

(2) Indicators of quality: 

• A supportive school climate which is also visible in the 
classroom. 

• * Clear statements of school goals and expectations which are 

shared by students, school staff, parents, and other 
community representatives. 

• Other components of effective schools as indentified in the 
literature, e.g., 1) school mission; 2) leadership; 3) school 
climate; 4) high expectations; 5) instruction* 1 improvement; 
6) assessment. 
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(3) Indicators of equality: 

• Attendance rates for at-risk students whi^h equal or exceed 
those for the entire schooL 

• Retention and completion rates for at-risk students which 
equal or exceed that for the entire schooL 

• Progress toward educational goals and expectation for at- 
risk students which is equal to that made by all students. 

To employ such indicators in the identification of effective 
schools, it is important to remember that such data must be available 
on a disaggregated basis. Effective schools must collect, record, and 
retain quantitative and qualitative data in a fashion that supports 
longitudinal analysis of the performance of individual students and 
groups of students. Such analyses would include, for example, data 
disaggregated by race, ethnicity, gender, socioeconomic status, and 
grade level. Anything less and there is no way to ascertain whether 
sustained progress is being made for all students. 

Schools that provide effective schooling are humane and 
creative problem-solving institutions that engage students in 
academic learning processes which enable them to become capable of 
full participation in a free society that needs intellectually capable 
citizens. 
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Quality Indicators for Monitoring Equity 

Ramsay Selden 
Council of Chief State School Officers 

I am the director of the effort by the Council of Chief State 
School Officers to develop better indicators for education. The part of 
that job that is politically popular right now and the one that 
generates interest in the press is the issue of state by state 
comparisons and comparative achievement testing. It is, however, a 
small part of what I believe to be the very important problem of 
developing better indicators for monitoring educational equity. We 
should have better indicators, but we don't, because it is not the 
primary task that is before a group of educators like this one. Your 
primary task is making schools work for underachieving students. I 
think, however, that the development of educational indicators is a 
supportive task that can make your efforts work better and be more 
successful. I am in the business I'm in because I believe in 
indicators and their value. 

We need better information on how schools are functioning for 
underachieving minorities. I would like to explain how that would 
work, in a sort of idealistic sense, and then tell you, from state 
policymaker's perspective, where we stand in terms of getting the 
information we need. 

This model of the distribution of excellence in the school 
system was developed by Jeannie Oakes of the Rand Corporation. I 
liked it before this meeting, but now I like it even more, because it 
pulls together many of the things that have been discussed during 
this conference. According to this model, there is a process that 
results in inequitable outcomes in the school systems in this country. 

This process begins at the elementary school level, where the 
extent of a child's interest and achievement leads to placement in 
one of two routes at the junior high level. If the elementary school 
child has an opportunity to develop interests and he demonstrates 
achievement, that child is much more likely, according to the 
research, to be placed in high school academic prep programs, or 
even initial high school courses at the junior high level. 
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That placement leads to the high school college prep program, 
which is characterized by more academic courses, courses that are 
sequential in nature, courses that accumulate and build on one 
another, and courses that present advanced-level content. The 
advanced courses stimulate additional interest and persistence in 
studying academic subject matter beyond minimum requirements. 

Down the other route, the elementary child /ho does not 
experience success in academic achievement early in elementary 
school loses interest* and tends to be placed in intermediate or junior 
high school programs that are remedial in content and 
practical in orientation. That placement leads them to high school 
programs that are vocational, general, or remedial, and that are 
characterized by fewer academic courses. Generally, courses that 
don't build upon one another and that address lower level content 
are not sequential. These courses promote a lack of interest* 
persistence, and further achievement on the part of the students 
enrolled in them. These students are much less likely to overcome 
that experience and be interested in going on to the high school or 
higher education levels. 

Although these two paths are very logical and feel intuitive, 
the model was constructed by synthesizing research. The snowball 
effect described above is what really happens in schools. This is how 
kids get tracked down one route or the other. I think one of the 
most troubling aspects of this problem is the clear recognition that 
the process starts early, at the elementary school level. 

Based on her model, Oakes developed a set of indicators to 
show the extent to which schools are engendering that routing and 
those patterns of success for minority students and for women. The 
distribution of excellence for minority students and for women were 
both considered during the development of the indicators. 

Basically, the model includes two types of indicators, indicators 
that measure outcome or performance, and those that measure the 
processes within schools that characterize how schools are serving 
different kinds of students. 

The outcome or performance indicators, which are evaluated 
over time, incorporate achievement data by SES, race, and gender. 
The data need to be coded this way in order for overall patterns of 
achievement to be analyzed and monitored. The important thing is 
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that the data be broken down into all three categorical groups at the 
same time ? not just by race without accounting for gender or SES, 
There are various differential inequity effects within racial groups 
when they are considered by gender and by SES, 

There are a variety of possible outcomes, for example, what 
students study and learn (achievement in basic skills or subject 
areas), the courses they complete, and their interests in and their 
rate of high school completion. These outcomes can be monitored 
comparatively and over time in order to determine how equitable 
services in the school are. All aspects of the outcome indicators 
should be recorded by SES, race, and gender, so that cross-tabulation 
by groups can be shown. Means within groups should be looked at, 
and because the comparison of means tends to mask the fact that 
there is a great deal of variation within groups, the distributions 
across groups also should be considered. 

The process indicators were developed to show if schools are 
delivering their services equitably. Looking at both the indicators of 
school and instructional process, then, gives educators a picture of 
how effectively ard equitably programs are working, and signals 
areas in need of strengthening. For instance, a first step might be to 
monitor achievement in grades four, eight and eleven by means, and 
by group, and by the distribution of groups across quartiles. This 
gives a sense of the extent to which students in racial and ethnic 
groups are being distributed across the performance range. The data 
need to be broken down by SES, race, and gender, and cross* 
tabulated. The school might also keep track of the development of 
student interests, student confidence, and parental encouragement at 
each of the grade levels. As group differences in the variables 
emerge, the schools can then detect those differences and respond to 
them. Attitudes also should be monitored, especially those related to 
further study in the academic areas. That monitoring in this 
example, would begin in the fourth grade and be repeated in the 
eighth and tenth grades. 

At the elementary level, attention to process mipht mean 
keeping track of the instructional time spent in various subjects, 
again by SES and racial makeup of the school, and amount of 
homework. We should look at participation by race and SES in high 
school courses, and the ratio of course enrollments to the proportion 
of groups in the school population. How disproportionate is student 
representation in courses like calculus or academic chemistry? 
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Next to be considered in Oakes* model are resource indicators within 
the school. These classify the school by its racial and SES makeup, 
and look at things such as for-pupil expenditures, teacher salaries, 
pupil -teacher ratios, and class size. Are the schools within a district 
different in a way that is related to the socioeconomic and racial 
makeup of the school? At the elementary level we should keep track 
of the instructional time spent in various subjects, again by SES and 
racial makeup of the school. We should look at participation by race 
and SES in high school courses, and the ratio of course enrollments to 
the proportion of groups in the school population. How 
disproportionate is student representation in courses like calculus or 
academic chemistry? What is the distribution of curriculum 
resources like laboratory facilities in instructional processes? 

Schools must have a plan for raising minority achievement. 
Plans to boost achievement should contain supplemental programs, 
extracurricular activities and programs, staff development programs, 
curriculum development efforts, and special guidance programs. 
Data collected for process indicators can help keep track of these five 
elements. 

The Chief State School Officers are working on two projects that 
are helping us determine the extent to which we will be able to 
monitor this kind of comprehensive information at the state level. In 
one project we are looking at the core database in education, which is 
collected by each state. We find, from the demographic indicators on 
a state level, that states do not have standard data on the locale of 
their schools, whether they are urban, suburban or rural, or what 
types of neighborhoods these school serve. School enrollments are 
not cross- tabulated by race or sex. We have no standard SES 
indicator for schools that is collected and used by the states. 

The national core database for outcomes shows that state-level 
data that classifies graduates by school and racial or ethnic group is 
not generally available. Some states have it, some don't. We don't 
have standard dropout data at the state level. We don't have 
rar.dard dropout data at all; most states don't collect dropout data 
by student characteristic. 

In the second project, undertaken in collaboration with the 
National Science Foundation, we are looking at the math/science 
indicators that are collected *.. ( d used by each state. Thirty-five to 
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forty states have achievement data in subjects like math, science, 
and reading. We are currently studying whether they report that 
data by race, sex, and SES. Similarly, four to seven states collect 
attitude and interest information. We are finding out now if they 
report that by student group. Seven states keep track of the post- 
secondary majors selected by students and they break that down by 
racial or ethnic group of the student-but only seven states. One 
state monitors time spent in different subjects in the elementary 
school by group of students served-but only one state. Ten states 
monitor student enrollment by subject and ethnic group. 

Thirty-eight states -I think this is kind of surprising— only 
thirty-eight states regularly keep track of the ethnicity of their 
teachers. Forty-eight keep track of teacher assignments and 
certification. That information could be broken down by the kinds of 
students those teachers serve. We don't know yet, but I think that 
relatively few states are doing that now. Only four states observe 
teacher performance. If we wanted to know if teachers perform 
better with advantaged versus disadvantaged students, we would 
not have the information to do so. 

Only ten or eleven states keep track of the amount or type of 
professional development programs provided to teachET?^ only two 
states keep track of the extent to which teachers are involved in 
voluntary professional enhancement activities. Twenty-three states 
have the potential of monitoring teacher knowledge of their subject 
matter. Thirty-six states keep track of pupil-teacher ratios and 
fifteen or sixteen record the classroom resources that are provided. 
We don't know yet if any of these data are broken down by the type 
of student served. 

The main point is that the rudimentary information a state 
level policymaker needs to determine if his school system is meeting 
equity concerns isn't even available. We don't have the basics , let 
alone the breakdowns by student type that we need to monitor these 
issues. What I see as the first and most important problem is that 
people simply don't have a strong model and rationale for monitoring 
these equity concerns, I think this is where Oaks' model of the 
distribution of excellence has made a contribution. 
We intend to get our people to do some thinking about how they 
might apply such a model within their individual states. 
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Secondly, we are running into tremendous resistance to change. 
Existing data are sacred, and there is a horrendous inability on the 
part of state and local school people to change it They think if they 
redefine the data or add information, somehow their continuity with 
the past will be broken. I don't know if this is the issue in every 
case, or if people just don't want to do additional data collection. 
There is a tremendous, tremendous resistance to change. We 
proposed last year that states start collecting standard demographic 
data on school enrollment in every school in the country, including 
the sex and race breakdowns within each school—not by grade, but 
by the school as a whole. We ran into tremendous resistance on the 
part of state data collection people. 

There is also something of a money issue. I think it is less of 
an issue when it is a matter of expanding and tuning the existing 
data collection system, in which we now invest relatively little. A 
little bit of money would help us all offset some of the additional 
costs. 

In order to ta^e the recommendations that people have been 
making in this conference, the recommendations have to be 
translated into objective data that can be used to keep track of how 
the school systems are xoing in regard to underachieving minorities. 
We aren't very well equipped to provide and use that data right now, 
at least among our constituents. There is a need to reformat data 
collection, and in many cases, add to the data collection that already 
takes place in schools. People have to accept that the value and 
importance of the information is worth the restructuring process, 
even if it increases the burden of collecting the data. We have a lot 
of work to do. 
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THE ROLE OF TESTING AND EVALUATION 



Persistence and Patience 

Todd Endo 
Director of Research and Evaluation 
Fairfax County Public Schools 

Commitment to the academic success of black students 
clearly requires a sensitivity to the need for resolving 
these crucial cultural conflicts between school and 
community. Persisting in the face of these conflicts is 
more than many students, who see themselves as 
powerless, can manage* The teacher must affirm those 
behaviors needed to demonstrate achievement in the 
larger academic community, while simultaneously 
understanding the cultural and learning style diversity 
that the minority student brings to the learning 
environment. 1 

Like the teacher and the student who must learn to persist in 
the face of conflict and difficulty, the school system that seeks to 
improve the achievement of minority students must persist as well. 
Brilliance, creativity, and political savvy are all desirable. 
Technically sound analysis and use of data (even disaggregated data) 
are necessary but not sufficient steps to be taken. Persistence, 
patience, and just plain plugging along are mundane but vital 
ingredients that spell the difference between a good plan and a 
successfully implemented program, 

The Fairfax County Public Schools is in its second full year of 
implementing a series of activities designed to improve the academic 
achievement and aspirations of minority students. The Fairfax 
program features school-based planning, implementation, and 
evaluation to match the identified nesds of the students, 
implementation, and evaluation to match the identified needs of the 
students, skills of the staff, and potentially effective strategies for 
improvement in individual schools. School-based planning is 
conducted within a central mandate and framework. Other key 
ingredients are a sufficient timeline to develop and implement 
activities, additional resources of money and time, intensive support 
and assistance from central and area offices, careful monitoring and 
feedback, and above all dogged persistence and patience. 
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As the program has developed, there has been a continuing 
struggle to keep it on course. The temptations have been to speed up 
the process, to direct more and nurture less, to treat all schools like 
identical parts of the machine, to look for blueprints for the quick fix ? 
to simplify the expected outcomes to standardized test scores, and to 
wash away the emphasis on minority student achievement in the 
tide of competing priorities and crises. 

By the end of the first full year of implementation in 1985-86, 
modest improvement in minority students' achievement occurred. 
However, being in only the second year of implementation, we must 
persist through the full stages of implementation and be patienr 
before judging the success of Fairfax's minority students' 
achievement program. 

This paper first describes briefly the Fairfax program* Then it 
develops in more detail selected, important aspects of the program. 
Finally, the paper discusses some dilemmas, tensions, and unresolved 
issues in the program. 

Background for the Fairfax County Program 

Planning 

In the summer of 1983, the Fairfax County Public Schools 
established a staff study group 2 which analyzed 1982-83 school 
achievement data to determine the status of minority students' 
performance. The study group purposely chose to examine regularly 
collected data that were broader than just standardized test scores. 
The eight indicators of achievement that were examined included: 

• retentions in grade 

• enrollment in higher level and lower level courses 

• grades 

• placement in special programs 

• attendance 

• dropout rates 

• post-secondary school educational plans 

• standardized test scores 

Data on these eight indicators of achievement were analyzed by 
four ethnic groups: white, black, Hispanic, and Asian. Inclusion in a 
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specific ethnic group was determined by the response of parents on 
the student's enrollment card to a question that used the categories 
provided by the U.S. Office of Civil Rights, 

The study group concluded that the achievement of black and 
Hispanic students was unacceptably low and below that of white 
students on nearly every indicator examined. 

To address this identified problem, the study group concluded 
that the most promising strategies were directly related to the 
factors identified in the research on effective schools and effective 
teaching, namely: administrative leadership, high expectations for all 
students, defined instructional objectives, an orderly environment 
for learning, systems for regularly monitoring student learning, and 
parent and community support. The study group further concluded 
that improvement in minority students' performance was achievable 
as part of a long-range plan for school improvement that would 
benefit all students. Specifically, the study group proposed the 
following elements of a successful plan: 

* Establishment of a major, long-range school board 
priority to improve minority students' academic 
achievement and aspiration 

* Leadership from the superintendent and other top school 
system leaders through providing direction, announcing 
expectations, generating momentum, and providing 
visible support and attention 

* Development of local school improvement plans which 
address the achievement and aspirations of minority 
students within guidelines established by the school 
board and superintendent 

* Commitment of additional resources (time, money, and 
staff) and the reallocation of some existing resources over 
an extended period of time 

* Development of a systemwide plan for program and staff 
development based on the effective schools' research and 
the sharing of successful strategies for improving 
minority students' academic achievement 
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* Provision of additional evaluative tools and training to 
teachers and principals to monitor student achievement. 

Implementation of the Program 

Although the implementation of the minority students' 
achievement plan has evolved since its beginning in 1984-85, it 
basically has stayed true to the vision expressed in the initial staff 
report. The balance between direction, srpport, and monitoring by 
the school system and the development, implementation, and 
evaluation of school-based plans by school staffs has been critical 3 

Central Direction 

The school board, superintendent, and top level staff 
persistently announce that the improvement of minority students' 
achievement is a major school system priority and that they expect 
the multiple indicators of achievement to show improvement over a 
number of years. This priority and direction are reflected 
consistently in the superintendent's annual operating plan for the • 
school system, the divisionwide plan for the improvement of 
minority students' achievement, publications, speeches, remarks at 
school board meetings, memos, and a varietv of other means of 
communication. 

The divisionwide plan issued at the beginning of the 1985-86 
school year stated a series of objectives and evaluation strategies 
that relied on the eight indicators of student achievement used in the 
study. At the same time, the superintendent emphasized that while 
he planned to monitor results in terms of student outcomes, he 
believed that the best place for program planning and 
implementation was at the school level. He also indicated that the 
role of the central and area offices was to "support both divisionwide 
emphases and school-community identified objectives." This focus 
on school-based planning was also contained in the superintendent's 
annual operating plan. He directed the schools to develop, implement, 
and evaluate plans according to an establisned time schedule and 
committed systemwide resources to support the schools' efforts. 
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School-Based- Planning 



The central element of the program is school-based planning. 
The process involves an annual cycle of activities: 

* Development of final objectives, strategies, and 
evaluation plans in the early fall 

* Submission of plans by the end of October 

* Review of plans and return of individualized written 
responses to principals by the superintendent in 
November 



* Implementation of the plans throughout the year 
beginning in September 

* Collection and analysis of data at the school throughout 
the year 

* Review of activities, progress, and concerns by the area 
superintendents and the deputy superintendents at mid- 
year 

* Submission of an annual evaluation report in June that 
reports results and discusses implications for the next 
year's plan 

* Review of evaluation reports and return of individualized 
written responses to principals by the superintendent in 
July that include advice and requirements for the next 
year's plan 

* Repetition of the cycle 

Often merely a paper exercise, this process gains vitality 
through a number of strategies. First, school system leaders visibly 
and repeatedly announce their intentions and expectations. Second, 
they follow up by personally showing specific interest by responding 
to individual school plans and reports, participating in mid-year 
review meetings, visiting schools, and discussing evaluation reports. 
Through these interactions, they encourage collaborative planning by 
members of the school community, recognize differences in plans due 
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to differences in school circumstances, encourage changes in plans if 
the situation or data calls for them, and support continuity in plans if 
all is going well. Third, central and area offices provide extensive 
and intensive support and assistance. 

With the steady stream of new priorities that flow across the 
experienced principal's desk, it is no wonder that his or her typical 
response to another new priority is "this too shall pass, 11 Visible 
persistence is required to convince principals that the minority 
students' achievement priority will remain. The school system 
leaders reinforce the message that they mean business by 
committing the resources and giving the assistance necessary to 
enable schools to succeed. 

Persistent reinforcement is also needed to convince school staff 
that their ideas are sought and that good school level plans will be 
approved. Many school staff are in the habit of trying to guess what 
the superintendent wants. Top-down management is part of the 
system culture. Principals, with good reason, often greet the call for 
bottom-up initiative with skepticism. 4 

Schools developed their first plans in the fall of 1984. Because 
of the late start, the first full planning and implementation cycle was 
completed in the 1985-86 school year. 

Individual school plans addressed one or more of the eight 
indicators specified by the school system. In addition, they 
developed objectives and activities determined by the school. These 
included quantity and quality of student writing, reduction of 
disciplinary problems, more parental involvement, greater 
participation in school activities, and progress through reading text 
books. 

System wide Support and Assistance 

The school system supported its plan with action that gave 
credibility to its claim that minority students' achievement was 
indeed an ongoing priority: 

• Beginning in 1985-86, money was set aside to fund 
school-based proposals. This strategy not only provided 
additional resources to schools but also reinforced the 
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notion of school-based plans. The total available funds 
for grants has increased steadily to meet demand. 

Also in 1985-86, a coordinator was hired and resource 
teachers were provided to manage the program and 
provide assistance to schools. 

To help schools plan and implement, written models were 
provided, workshops were conducted, and individual 
assistance was given. 

To help schools monitor and evaluate the effects of their 
strategies, resource manuals were provided, workshops 
were conducted, group and individual assistance was 
provided, and on-going follow-up was provided at the 
individual school level* 

Systemwide staff development programs were offered to 
volunteer school staff* Examples included tue Teacher 
Expectations and Student Achievement (TESA) program 
and the Classroom Management Training Project begun in 
the 1984-85 school year. These programs reinforced the 
idea that strategies to improve minority students' 
achievement can be integrated with strategies for school 
improvement* 

Centrally based activities were conducted to pursue the 
system's direction and to support school-based activities. 
Examples included a review of procedures to select 
students for special programs, training for guidance 
counselors, overall coordination of the program, review of 
proposals for school-based grants, and initiation of some 
instructional programs. 

Related initiatives were linked to the minority students' 
achievement priority. For instance, for the 1985-86 
school year, the school board approved a proposal written 
by a group of principals to allocate additional resources to 
schools with a high proportion of students with special 
needs. In addition, in the spring of 1985, the school 
board approved the superintendent's recommendation 
for a new priority r 'to develop a system for school-based 
management. 44 
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The provision of supplemental resources was crucial to 
convince schools that the school system was serious. Perhaps more 
important were the individual attention given to schools and the 
accompanying message that school -based efforts were indeed 
supported. For rhetoric to become reality, persistent effort was 
required as evidence. 

School System Monitoring 

Based on the original staff committee report, an annual report 
on minority students' achievement is issued each year in the fall* 
The report provides system level trend 4ata by ethnic group on each 
of the eight original indicators* This report is presented at a school 
board meeting and publicized through the media* 

In addition, analyses are conducted to answer specific 
questions about subsets of schools or students* These analyses are 
provided to the requestor on an ad hoc basis* 

Moreover, area superintendents monitor progress through their 
related planning process. As mentioned earlier, they review the 
individual school plans and evaluation reports. They also are 
provided school level data on their schools for use in supervision. 

Schools are provided data on their students using the same 
indicators and ethnic group classifications used for the systemwide 
report. They use these data for planning, evaluation, and reporting 
purposes. 

Finally, a community advisory board reviews the process and 
progress and gives advice to the superintendent. 

Results to Date 

School plans were developed during the 1984-85 school year 
and the first full year of implementation was the 1985-86 school 
year* The annual report issued in October 1986 provided trend data 
for four years culminating in the first full year of implementation. 
The summary of the report stated: 5 

During 1985-86 the standardized test results showed 
improvements, especially for black students* Of all ethnic 
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groups, black students made the greatest gains on the SRA tests 
over the last four years* In grades 8 and 11, black students 
have made gains each year, increasing cumulatively by as 
much as ten percentile points in grade 11, On the Virginia 
Minimum Competency Test, the percentage of black students 
who passed the tests increased substantially in 1986* 

In addition, the percentage of black students scoring above the 
50th percentile increased by three percentage points over the three 
year period. Other minority groups improved on this measure 
slightly, while the performance of white students declined slightly. 
The percentage of black students scoring below the 20th percentile 
declined by four percentage points over this same three year period, 
while the performance of all other groups stayed the same on this 
measure. 

On the other seven indicators of achievement, the message was 
mixed. Placement of black students into programs for the 
emotionally disturbed and mildly mentally retarded showed steady 
declines over the four year period. The intention of black students to 
attend a four year college steadily increased* On the other indicators, 
no strong trends were noticeable. The report concluded: 

Clearly, minority student achievement is not at the levels it 
should be. While the first full year data in this report show some 
progress, justifying the Board's long-term commitment, only in the 
long term will minority achievement reach satisfactory levels. 6 

Next Steps 

Fairfax is now in the second full year of implementation of its 
minority students* achievement activities. For this year and the 
foreseeable future, reinforcement of the existing activities is planned. 
We expect that persistence will pay off. 

Jn addition, some new activities are planned that reflect a more 
advanced stage of development For instance, promising school- 
based practices will be identified and described. In printed form and 
through personal contact, school staff will share these practices with 
their peers* Perhaps grants will be given to schools so that they may 
adopt and adapt promising practices developed by others* 
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Also, because the program is approaching the end of the second 
full year of implementation, longitudinal data will be available on 
students. For example, students from the four major ethnic groups 
with similar achievement in grade 6 will be tracked on subsequent 
indicators of achievement, such as test scores, enrollment in higher 
level and lower level courses, grades, and placement in special 
programs. 

In the next few years, the minority students' achievement 
emphasis will be integrated with and reinforced by other major 
school system initiatives. Clearly, the emphasis on school-based 
planning will be reinforced by the school board and superintendent's 
priority to develop a school-based management program. As part of 
this program, a new process for selecting, training, assigning, and 
evaluating principals and other school-bar * administration is being 
developed This new process should help clarify the role of all 
administrators and enhance the possibilities for success of both 
school-based management and the minority students' achievement 
emphasis. 

Also related to the minority students' achievement emphasis is 
the school system training based on The Skillful Teacher? a generic 
instructional and supervision model developed by Jon Saphier, This 
model provides the framework for the instructional focus expected of 
school-based administrators and teachers, and is integral to the new 
performance evaluation program being developed in Fairfax. The 
cumulative effect of these initiatives will be to enhance the role of 
school-based staff to design, implement, and evaluate improved 
instructional strategies with a particular emphasis on minority 
students. 



CONTEXT AND ASSUMPTIONS OF THE FAIRFAX PROGRAM 

The Context is Fairfax County 

The process for improving the achievement of minority 
students described in this paper may be suitable only for Fairfax 
County. Just as there is no one best educational system, there is no 
one best strategy for improving the achievement of minority 
students. 

25 
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Fairfax County is a large, metropolitan school district in which 
minority students represent about 22 percent of the enrollment. 
Fairfax has sufficient staff and financial resources to support 
minority students' achievement programs as well as most other 
identified priorities. Most students achieve well and the public is 
generally happy with the school system. Standard operational 
functions relating to personnel, budgeting, accounting, purchasing, 
planning, and transportation generally happen routinely. A standard 
framework for curriculum and instruction is in place. Basically, this 
is so because the school-based staff and centrally -based staff are 
competent and have put into place systems to handle the routines. 
Though not without crises, Fairfax can afford to build on a solid 
foundation and plan carefully selected strategies for improvement. 

Assumptions Underlying the Fairfax Program 

The Fairfax program is built on many explicit and implicit 
assumptions, which rely on much of the research on effective schools 
and school improvement. However, competing models are also based 
on research* 8 Without discussing which model is more true, I will 
stale some of the assumptions of our program: 

• The school is the basic unit of school improvement, 

• The school board and superintendent should establish 
priorities and give general direction, but a school staff is 
in the best position to develop the specific plans to enable 
its school to move in the desired direction, 

• The emphasis on minority students' achievement is not 
an add-on and should be integrated with systemic efforts 
to improve instruction, 

• There is no magical solution or panacea. The task is to 
put together a set of available ideas and commit the 
system to implement the ideas with sufficient resources 
over an extended period of time, 

• The improvement process takes time, 

• The improvement process requires varied strategies to 
provide support and supervision. 
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The remainder of this section of the paper will expand on each of 
these assumptions. 

The School is the Paste I nit for Improvement 

Fairfax recognizes that the desired improvement at the school 
system level is composed of improvements made school -by -school 
and teacher -by -teacher. In addition to systemwide activities, great 
efforts must be expended in each school. 

The focus on the school implies a recognition that the whole is 
greater than the sum of its parts. That is, while the individual 
teacher is important, the school is more than a collection of 
individual teachers. In fact, there is a need to break down some of 
the isolation that teachers and administrators feel, to develop more 
collegia! and collaborative relationships, and to develop a sense of a 
common school mission. 

From this perspective, a school system could impose a uniform 
program and then implement it school by school, Fairfax took a 
different lack. 

School Staffs are in the Best Position to Plan and Implement 
Activities within Their School 

This statement contains within it a number of further 
assumptions. First, because each school is difterent, the school 
system should be careftu what it demands that all schools do. The 
superintendent has stated that he wants to hold principals 
accountable for results and give them flexibility in terms of the 
means. 

Second, most principals and teachers have sufficient skills, 
interests, and ideas to be entrusted with the development of school- 
based plans. In most schools, the principal and teachers can learn 
even ;c the central office is not teaching. Just as higher expectations 
for students are desired, the assumption is that high expectations for 
school staffs will lead to higher achievement. 

Third, if the staff is more involved in making program 
decisions, their commitment to the effective implementation of these 
decisions will be greater and the results for students will also be 
greater. 
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Central and area office staff tailor their support and assistance 
to the needs of the individual school and principal, vVhiJe some large 
group instruction is given, most activities are designed Tor small 
groups of likeminded staff or for individuals. This strategy is similar 
t a teacher designing total class, small group, and individualized 
instruction. 

Minority Students' Achievement is Not an Add-on 

Since there is a tendency to view a new initiative as an 
addition to the existing program, Fairfax is emphasizing that minority 
students' achievement activities should be integrated into the on- 
going instructional program, not added on to it. Therefore, Fairfax is 
encouraging improvement in the regular functions of a school; for 
instance, counseling students into existing higher level courses, 
better monitoring of individual studeiu progress, involving more 
parents of minority students in the ongoing life of the school, 
improving teacher expectations, examining screening procedures for 
special programs, and focusing on writing or thinking skills across 
the curriculum. Fairfax is not encouraging greater grouping of 
minority or underachieving students or pulling out identified 
students from regular classes for special instruction. 

There is No Magical Solution or Persistence Pays Off 

The superintendent has emphasized that there is no magical 
solution to the problem of improving minority students' achievement 
and that only a persistent effort over a number of years will succeed. 
A review of what the school system and each of the schools are doin^ 
reveals no dramatically new instructional ideas. The ideas are good 
but were culled from personal experience and reflection, discussions 
with colleagues, and insights from external publications and 
workshops. What does contribute to success is the persistent 
attention paid to planning, implementation, and evaluation. 
Persistent attention means that the improvement process takes time. 

The Improvement Process Takes Time 

The downfall of many new initiatives is the belief that policy 
decisions are self-implementing or are easy to implement. By their 
statements and actions, the school board and superintendent are 
attempting to resist the pressure to speed up the process. The 
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superintendent has emphasized the need for a long-term persistent 
effort that will yield slow but steady improvement. On many topics 
he has been quoted to say, "Just because we've planted the seed and 
it's germinating doesn't mean you can keep picking up the plant and 
looking at it every day." The school board considered the annual 
report, presented in October 1986 after the first full year of 
implementation, to be a progress report, not a summative report on 
the program's successes or failures. In the mid-year reviews, the 
deputy superintendents stressed that the priority will remain for the 
foreseeable future because the improvement process takes time. 

A second dimension of time is the "life space" needed by all 
staff, but especially principals and teachers, to plan, implement, and 
evaluate school board plans thoughtfully. A principal's and teacher's 
day is full of countless important and mundane events. It is too 
much to expect that school staffs will plan effectively on top of 
everything else they do. Time as a resource must be provided. This 
kind of time to plan, implement, and evaluate can be provided in 
three ways. One is to take away other tasks (e,g„ some competing 
priorities, some unnecessary meetings, some externally or internally 
imposed expectations, some paperwork), A second way is to provide 
substitute days and days before or after the contract year, A third 
way is more a state of mind than real time. We assume that when a 
staff believes it is the creator and owner of the plan, it willingly 
commits additional time to do the right job and to do the job right. 
When faced with the task of implementing a school system mandate 
for which it feels no ownership, we assume that a school staff tends 
to devote only enough time to comply. 

The Improvement Process Requires Varied Strategies to 
Provide Support, Assistance, and Supervision 

Besides time, the school-based improvement process requires 
support from the central and area offices. As described in the 
section in \ystemwide support and assistance," this support has 
taken the form of money, time for school staffs to plan, staff to 
coordinate efforts, and resources to assist school staffs plan and 
implement better. 

Just as assistance is tailored to the individual school, so is 
supervision. Through the monitoring activities, some schools are 
recognized as being on the right track. For them, little change is 
demanded and praise is given. Deficiencies are noted in other 
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schools' efforts. The area superintendent supervises these schools 
more regularly, giving more explicit instruction and direction. 

THE ROLE OF THE OFFICE OF RESEARCH AND EVALUATION 

While many departments of the school system contribute to the 
effort to improve minority students' achievement, this section 
focuses on the role of the Office of Research and Evaluation (ORE) 
because the activities of this office illustrate most of the basic 
components of the program and because I know the activities of ORE 
best. Much of this section could also be written about the activities 
of the coordinator of minority students' achievement, the area 
minority student achievement resource teachers, and other area and 
central departments* 

The Office of Research and Evaluation has played a central role 
in monitoring the achievement of minority students and in providing 
support and assistance to schools* Some ORE staff monitor the 
indicators of achievement for the school system, for identified groups 
of schools, and for individual schools. They provide data and 
analyses of data to decision makers, produce reports on the academic 
progress of minority students, and discuss the implications of the 
findings for program activities* 

Other staff in ORE provide support and assistance to individual 
school staffs in the development and implementation of school-based 
plans* The process for support and assistance is tailored to the needs 
of the individual school staff and, thus, is time consuming* In its 
highest form, and ORF staff person meets with the principal and/or 
member of the school staff to plan a needs assessment, discuss 
possible elements of the school plan, propose alternative evaluation 
strategies, and review drafts of the school plan* In an intermediate 
form, the ORE staff provides intensive assistance to schools in the 
development of their evaluation plans, the implementation of the 
plans (including methods of collecting and analyzing data), and the 
writing of the evaluation reports. 

In its most extended form, ORE has conducted two year projects 
with volunteer schools. These are designed to assist school staffs 
increase their capacities to identify, clarify, analyze, and use factors 
within the school in order to develop, implement and evaluate a long 
range, comprehensive plan for improving student achievement with 
a special emphasis on minority students. 9 The framework for this 



188 30 



effort drew freely from the experiences of the staff, from the 
writings of Fullan, Hall et aL, Little, and Joyce and Showers on school 
improvement, leadership, and staff development, and from a variant 
of effective school projects across the country. 

The major explicit tasks of the first year of the project were to 
help school staffs conduct a needs assessment at the school and 
develop a plan to address these identified needs. Behind these 
explicit tasks were two implicit but necessary tasks. These were to 
develop both readiness 10 in the staff to "own" the project and the 
staff skills in planning. 

The formal activities of the first year included monthly 
meetings of the school-based planning team and two full day 
workshops on the planning process and planning objectives, 
activities, and evaluation strategies relating to identified needs. In 
addition, many informal meetings were held among the school staff 
and between members of the school staff and ORE staff. The role of 
the ORE staff was to initiate and develop, in conjunction with the 
principal, most of the planning activities for the monthly meetings 
and workshops; to circulate appropriate articles and otherwise link 
the school-based planning teams to useful resources; to develop data 
gathering instruments and analyze some of the needs assessment 
data; and generally to serve as a "sounding board" and consultant to 
the principal and teacher chairpersons. 

By the end of the first year, each school had developed a school 
improvement plan. In the process, the planning committees 
developed strong collegial relationships, a sense of school mission to 
guide classroom responsibilities, ownership of the planning process 
and the plans, familiarity with much of the research literature, and 
knowledge and skills related to data collection and analysis. 

The second year featured implementing the plan and 
evaluating activities. The major responsibility for the project shifted 
to the school staff. The ORE staff still conducted workshops and 
influenced events, but their role became more one of support than of 
leadership, of responder than of initiator. Major emphasis in the 
second year was on broadening the staffs repertoire of evaluation 
strategies and developing real use of selected strategies for school- 
based improvement. The repertoire included systematic use of peer 
and principal observations, teacher and student interviews, report 
card data, office records, basal test information, anecdotal records, 
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instructional grouping information, teacher and principal anecdotal 
records, instructional grouping information, records of attendance 
and participation, and various types of standardized and teacher- 
made tests. By the end of the second year, each school had 
completed the first year's implementation and the first report of 
progress, 

School-Based Evaluation is Not as Easy as It Sounds 

We were surprised by how hard it was for some schools to 
develop school-based evaluation plans that would help them 
improve their programs. Use of data by school staffs to help improve 
instruction for students did not occur just because the data was 
available and technical assistance was given. Our reflections on the 
reasons for this difficulty may offer insight into the change process 
and why it is necessary to persist over time. 

Talking about testing, a researcher once noted that everyone 
thought testing was important — for someone else. Teachers thought 
testing was important for principals, principals thought testing was 
important for the superintendent, and the superintendent thought 
testing was important for teachers. But few people thought testing 
was important for themselves. 

Similarly for evaluation, the prevailing image was hard to 
break. School principals and teachers generally had the view that 
evaluation was something done to them by someone outside the 
school usually with negative consequences. In planning evaluations, 
many of them initially focused on summative not formative 
evaluation, hard data not soft data, and pre-testing and post-testing. 
Perhaps because of the view that evaluations were negative 
accountability strategies, many principals tended to focus on 
activities they could control (e.g., number of staff development 
sessions held), rather than on outcomes for students or others. They 
tended to view evaluation as a game in which the object was to state 
objectives that could be accomplished. 

Aaron Wildavsky, a political scientist, wrote an article, entitled 
"The Self-Evaluating Organization/ 11 that addresses some of these 
same difficulties. Wildavsky reflects on his intellectual journey— "1 
started out thinking it was bad for organizations not to evaluate, and 
I ended up wondering why they ever do it. 1 He explores why 
evaluation and organization tend to be in tension: 
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Evaluation and organization may be contradictory terms. 
Organizational structure implies stability while the process of 
evaluation suggests change. Organization generates commitment 
while evaluation inculcates skepticism. Evaluation speaks to the 
relationship between action and objectives while organization relates 
its activities to programs and clientele, 12 

As part of a school organization, then, principals and teachers 
were, at the least, in tension over the call to evaluate themselves. 

It was only through persistent and intensive efforts that some 
progress was made in dealing with these tensions. Part of the 
success can be attributed to the distribution of a resource notebook 
for school-based evaluation, work-shops on the subject, technical 
assistance in small groups and individually, and more intensive 
effort such as that described in the previous section. Part of the 
success also resulted from persistent feedback that objectives should 
be stated in terms of outcomes, that use of non-test data is 
encouraged, that data derived from teacher journals are acceptable, 
and that non-attainment of objectives is more a signal to improve 
than to condemn. Much of the success also came from principals and 
teachers realizing that not only do good evaluation plans and reports 
gain external praise, but also that the data are useful to them as they 
seek to improve instruction. Finally, part of the success only came 
about because the ORE staff and the school staff persevered and 
eventually came to trust and rely on one another. Still, it would be 
foolish to say that self-evaluation is now the norm of the school 
system. 

The School Improvement Process 

What we experienced in the project reinforced our view that 
"change is a process, not an event/ 13 and that the process is a long 
and difficult one that requires more time and persistence than 
anticipated, A major reason for the length and difficulty of the 
process is the necessity for some basic changes to occur in the culture 
of the school, culture of the school system, and the perception of 
roles by principals and teachers. It is not easy for teachers and 
principals to move out of their isolation toward a sense of school 
community with a definite mission. It is not easy for teachers and 
principals to believe that they can make important decisions and to 
act on that belief by taking ownership of ideas and pursuing them. 
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It is; not easy for colleagues to develop enough trust to open up areas 
of uncertainty, to question oneself and others, to make mistakes in 
front ot others, to rely upon one another* It is not easy for a school 
staff to really believe that the central office is there to help them* It 
is not easy because control, compliance, hierarchy, isolation, and 
doing the thing right are stronger elements of the crlture than 
school-based decision-making, collaborative planning, collegiality, 
mutual problem solving, and doing the right thing 

Wildavsky and Judith Warren Little 14 discuss some of these 
difficulties and agree on some of the positive characteristics of 
improving and evaluating schools. Little describes "a norm of 
collegiality/' while Wildavsky states that "an extraordinary degree of 
mutual trust" is a requirement of a self-evaluating organization. Our 
findings support that a context of trust and collegiality is a 
prerequisite for the honest use of data for self-evaluation. 

Little also describes "expectations for analysis, evaluation, and 
experimentation: a norm of continuous improvement." She talks 
about the need for "aggressive curiosity and healthy skepticism" and 
concludes that "where analysis, evaluation, and experimentation are 
treated as tools of the profession, designed to make work better (and 
easier), and where such work is properly the work of the teacher, 
teachers can be expected to look to staff development to help 
provoke questions, organize analysis, generate evidence of progress, 
and design differences in approaches/ 15 Wildavsky agrees on the 
need for a "climate of opinion that favors experimentation" and 
skepticism and states as an example that "oiganization members 
would have to be rewarded for passing on bad news." 16 Merely to 
state these points is enough to realize how far away most schools and 
school systems are. However, as Little and we have shown, in some 
schools such a supportive culture exists. But progress is slow* 
Persistence and patience is called for. 

TENSIONS, ISSUES, AND DILEMMAS 

Bottom-Up and Top-Down 

This is purposely posed as a both-and statement. Minority 
students' achievement is both a top-down and a bottom-up 
enterprise. The trick is to know what to mandate from the top-down 
and what to encourage to emerge form the bottom-up. Basically we 
mandate from the top the expectation that minority students' 
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achievement will improve as measured by the eight indicators, the 
school-based planning process and format, and a monitoring system. 
At the same time we encourage local school initiative and creativity. 
Where it is difficult to know whether to operate bottom-up or top- 
down include the following situations: 

* Reviewers of plans and reports conclude that the efforts 
of the school are inadequate and can be improved. When 
should the principal be encouraged? When should he or 
she be directed? When should he or she be removed? 

* When is something deemed so good that it should be used 
more broadly? For instance, The Skillful Teacher training 
has been well received and next year all schools will be 
involved in the training. One consideration in the 
selection of this model was that it is generic and not 
prescriptive. Thus, within its framework, it allows great 
flexibility to accommodate individual teacher and school 
differences. 

* When is there a window of opportunity that must be 
used even if the system and the schools are not ready? 
For instance, at a given time, the political climate may be 
right, important actors in agreement, or additional funds 
available. 

There is a natural tendency and pressure to centralize this 
process. However, it is important Q£l to overcentralize* Although it 
may seem more efficient to direct schools to implement the same 
strategy, the effectiveness of such an approach is questionable. For 
those who monitor the program, it is easier to feel that you know 
what's going on if all schools are implementing the same program; 
and it certainly is easier to describe. The parents, community, and 
other external audiences can more easily understand a centralized 
program and thus may develop greater confidence in the effort. It is 
hard to persist against these pressures and tendencies, but it is 
important to do so. 

The Pace of Change 

The resolve of the school board and superintendent to expect a 
slow but steady improvement in minority students' achievement 
runs up against the pressure to show results quickly. The black and 
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Hispanic communities are understandably impatient The desire to 
show positive results in the local and national press is also 
understandable, as is the desire to compete with the neighboring 
school systems* Pressure to do it faster, cut corners, and be more 
efficient is hard to resist. It is easy to understand that Wood's 
readiness and maintenance phases of the change process frequently 
are short-changed. 

Researchers and the Department of Education have warned 
against cookbooks (Finn), recipes (D'Amico), and the list mentality 
(Barth). The warning is appropriate but difficult to heed. 

The Definition of Minority 

The focus of the program can be on different groups of 
students and the choice can reflect different purposes for the 
program. In Fairfax there has been continuing discussion of the 
alternatives. 

The initial school board priority emphasized improvement in 
the achievement and aspirations of minority students. In practice, 
the emphasis wa? placed on black and Hispanic students because the 
data indicated that Asian students were generally doing well. The 
priority purposely did not emphasize low achieving students in 
general or low achieving minority students in particular. The 
priority was intended to include the moderately achieving student 
who could do better a* well as the low achieving student. 

The continuing debate resulted in a change in the annual 
operating objective for 1986-87. This year's objective calls for 
"improving the academic achievement and aspirations of 
underachieving students, with emphasis on meeting the needs of 
minority students.*' 

Focus on Test Scores 

Test scores as a measure of student achievement are accepted 
by a variety of audiences. Researchers use test scores to judge the 
effects of a new program. The newspapers feature the results of the 
College Boards, norm-referenced tests, and minimum competency 
tests and rank school systems and schools on the basis of these. 
Parents, realtors, and the general public want to know what the test 
scores are. Yet, thoughtful principals, teachers, central office staff, 
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and researchers realize that test scores give only a partial picture of 
student achievement If we want principals and teachers to take 
ownership of school-based evaluation, we must support the use of 
more than test scores as a measure of student achievement. 

Fairfax consciously chose a variety of indicators of student 
achievement to assess the success of its efforts to improve minority 
students' achievement Spokespersons consistently speak about all 
the system wide indicators and the variety of school-based indicators 
in order to curb the powerful impulse to simplify the criteria to one 
only of test scores. It helps to use illustrations, such as the classic 
underachiever who has high test scores, uneven grades, and low 
attendance. It is also important to talk about what decisions are 
made as early as the sixth grade in order for a student to take and 
succeed in calculus in the twelfth grade. 

A more powerful reason to emphasize indicators other than 
test scores may be emerging from our data. Test scores are rising, 
but most other indicators are not showing much, if any improvement. 
This may reinforce the view of the sometimes cynical principal who 
said that he could raise test scores, if that was what was wanted, 
even though student achievement would not improve. Use of 
multiple indicators will give a more complete picture of the changes. 

Closing the Gap? 

Most of the recent discussion concerning minority students' 
achievement talks about closing the gap between the achievement of 
minority students and the achievement of non-minority students. 
Fairfax has tried to resist this characterization of the goal Instead, it 
has stated as its goal to "improve the academic aspirations and 
achievement of minority students." It has done this in order to 
compare minority students' achievement with itself or with the 
nation and not with other student groups in Fairfax. 

An example will illustrate this difference. If the pass rate of 
minority students on the state minimum competency test increases 
by five percent, is the improvement any less if the pass rate of non- 
minority students also increases by five percent? For this reason, 
charting the change in the percent of minority students who score 
above the national norm on the test is more important than 
comparing the mean scores of minority and non-minority students. 
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It would be fine if the achievement of minority students 
improved and the gap is narrowed in comparison with non-minority 
students. But, the program can succeed if minority students* 
achievement improves even if the gap is not narrowed. 

The Science and the Art of Education 

The prospectus for this symposium promised contributions to 
the creation of a science of education. Tm not sure my 
contribution will advance that cause. In my view education is 
more an art or craft than a science. Using creative analyses of 
available data is an important tool in the effort to improve the 
achievement of minority students and all students. But, in my 
view, data analysis is but one tool in the hand of a craftsman and 
not the most important one. My image of the good 
superintendent, principal, teacher, and even the director of 
research and evaluation is the master craftsman not the eminent 
scientist. 

I hope that no one attempts to replicate the "Fairfax model" in 
their situation. What I do hope is that readers will run the ideas and 
experiences presented here through the filter of their experiences, 
values, beliefs, hunches, and situation and shape what remains into 
something that makes sense to them. Then, persist and be patient. 

Footnotes 

1. As I was finishing this paper, a new booklet appeared on my 
desk: Learning to Persist and Persisting to Learn written by Bessie C 
Howard and published by the Mid-Atlantic Center for Race Equity in 
1987. It caught my eye first because its title contained a major 
theme of this paper-that is persistence is a crucial factor. The 
booklet captured my attention because it was about improving the 
academic achievement of minority students. As I read the booklet, I 
realized that much of what it said about the conflicts between 
minority students and the routines and expectations of the school 
system could also be said about the conflicts between principals and 
teachers, who are instructing minority students, and the routines and 
expectations of the school system. 

2, The background and impie mentation sections of this paper are 
adapted from three Fairfax County Public Schools reports. The first 
is "Minority Students 1 Academic Performance: A Preliminary Report" 
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issued in January 1984. The second is the "Report of the Advisory 
Committee on the Academic Performance of Minority Students in the 
Fairfax County Public Schools" issued in May 1984. The third is the 
"Report of Minority Students' Achievement for the 1985-86 School 
Year" issued in October 1986. 

3. A useful think piece exploring a similar view in more depth is 
contained in Philip Schlechty's chapter, "District Level Policies and 
Practices Supporting Effective School Management and Classroom 
Instruction," in Regina Kyle's (editor) book, Reaching for Excellence : 
An Effective Schools. Sourceboo k. Washington: U.S. Government 
Printing Office, 1985. 

4. A description of a bottom-up activity in Fairfax County and 
thoughts on how to encourage more such activities is contained in a 
paper, "Bottom-Up from the Top-Down," by Todd Endo, and published 
in Reflections , a journal of the National Network of Principal Centers. 
Copies can be obtained from the author. 

5. "Annual Report on the Achievement and Aspirations of 
Minority Students in the Fairfax County Public Schools," Office of 
Research and Evaluation, October 1986, Executive Summary. 

6. Annual Report, Executive Summary, 1986. 

7. Saphier, Jon and Gower, Robert, The Skillful Teacher 
Carlisle, MA: Research for Better Teaching, Inc., 1982. 

8. A very thoughtful description and analysis of various models 
for school improvement and the development of effective schools is 
contained in Larry Cuban, "Transforming a Frog into the Prince: 
Effective Schools Research, Policy, and Practice at the District Level," 
a report for NIE (June 1983). 

9. This section is a summary of a more extensive informal report, 
"Effective Schools Project Report, "written by the major developers of 
the project, Ann Cricchi and Mike Harrison of the Office of Research 
and Evaluation. 

10. The term, "readiness", is used in the context of the RPT1M 
(readiness, planning, training, implementation, and maintenance) 
model described by Fred Wood et al. in "Practitioners and Professors 
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THE ROLE OF TESTING AND EVALUATION 



Monitoring and Improving School Learning 

Walter Hathaway 
Portland Public Schools 

Introduction 

One of the greatest challenges facing American education in the 
second half of the 20th century has been the differences in 
achievement among students of different ethnic groups. Today, 33 
years after Brown vs. Topeka, and 22 years after the Elementary and 
Secondary Education Act of 1965 propelled our nation along the road 
to equitable and excellent education for all children, there are signs 
that the achievement of minority students is beginning to catch up 
with that of nonminority students on a nationwide basis. The 
remaining gaps in achievement test scores and other indicators of 
stuient performance, however, are still large and perplexing. 
Excellent and innovative educational assessment and progress 
reporting has and can help us respond to that challenge both by 
targeting instruction on specific student needs ard by focusing the 
attention of policy makers and other problem -solvers on the larger 
underlying issues. Dr. Ron Edmonds pointed to this promise of 
assessment in his seminal work on school effectiveness when he 
included "frequent assessment" as one of the key factors in 
improving schooling for all students but particularly for previously 
under-achieving minority children. (Edmonds, 1979) 

The Portland, Oregon school district has pioneered a number of 
improvements in achievement testing and test results reporting that 
have helped teachers improve the performance of all students while 
reducing the achievement differences among students of differing 
background. The Portland district also is one of those who have 
begun to use test and evaluation data disaggregated by ethnic group 
to identify and respond to the needs of previously underachieving 
minority students by improving programs, policies, and resource 
allocations. 
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This paper first summarizes some of th* national data on 
student achievement test results by ethnic groups. It then briefly 
describes some of the innovative features of the Portland testing 
program th?t have helped improve the effectiveness and equity of 
instruction. Finally it reports why and how the Portland School 
System has begun reporting and using disaggregated student and 
system data at each of the key levels in the system, what some of 
our initial findings have been, and what some of the effects of such 
reporting have been. 

The National Achievement Gap 

According to a recent report by the U.S. Congressional Budget 
Office, there is evidence from a variety of tests administered to 
students of various ages in different localities that: 

Recent years have seen a shrinking of the long-standing 
cl'fference between the scores of Black and nonminority 
students on a variety of achievements tests. The evidence 
pertaining to other ethnic groups is more limited but there are 
suggestions of relative gains by Hispanic students as well. 
While the change has been small relative to the remaining gap 
between the minority and nnminority students, it has been 
consistent from year to year and could prove substantial over 
the long run (U.S. Congressional Budget Office, 1986, pp. 74-75). 
(See Figure 1) 
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Figure 1 

Trends in Average Reading Proficiency for White, Black and 
Hispanic Students, by Birth Year 



Proficiency BlSCk 
Score 

300 ' 



Hispanic 



White 



?50- 



200- 



150 




Age 9 



n 1 1 1 1 1 1 1 M 1 1 1 1 1 1 ii 



Age 17 



Age 13 



* - Age 9 



Age n 



Ace 13 



Age 9 



J M | I I I | I I 4 | I I I j I I I 



I I I | I I I | I II | I 1 I I I I I 

1954 3959 1962 1966 1970 197 4 1954 1958 1962 1966 1970 1974 1954 1958 1962 1966 1970 197 
Birth Year Birth Year Birth Year 



SOURCE; National Assessment of Edu 
"AEP/Eductional Testing Service 1 



cational Process* The Reading Repo 
£85). Data Appendix. 



rt Card (priftcecor^ 



Although this Congressional Budget Office report raises cautions 
about the limitations of the data upon which this trend is based, its 
authors ciclude that the patterns observed seem genuine 1 

For example, they observe that in general, it appears that the 
average scores of students: 

* Declined less than those of nonminority students during the 
later years of the general decline; 

* Stopped declining, or began increasing again, earlier; and 

* Rose at a faster rate after the general upturn in achievement 
began. 

Recent evidence of progress toward diminishing the national 
achievement gap has, however, only begun to mute tlv 1 sad, national 
litany of indicators of the severe educational achievement 
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disadvantage of many minority children. It goes beyond 
achievement data and includes: 

* The dropout rate for minority students is as high as 80 
percent, versus a national average of less than 20 percent for 
nonminority students. 

* Minority students are often two to three or more grade levels 
behind on achievement measures. For example, the Southern 
Regional Education Board reports that on National 
Assessment Tests, "The reading achievement levels of Black 
eleventh-grade students is basically the same as for White 
seventh graders." (SREB, 1986, p. iv.) 

* In 1985 Black students had SAT Mathematics and Verbal 
scores that were 115 and 103 points lower than White 
students. Hispanic student scores were 65 and 67 points 
lower* 

* Black students are approximately three times as likely to be 
in a class for the educable mentally retarded but only half as 
likely to be in a class for the gifted and talented as White 
students, 

* Only about one-third of the estimated 2*7 million limited- 
English-proficient students aged 5 to 14 receive special help 
congruent with their linguistic needs. 

* Black and Hispanic students are two to three times as likely 
to be suspended or expelled and only half as likely to be 
enrolled in courses that lead to a college education. 

And the list goes on. 

Almost daily the Superintendents, Boards of Education staffs, 
and communities of school systems such as those represented in 
today's symposium find themselves searching for answers to the 
serious and complex questions of equity, efficiency, and excellence in 
education posed by such differences. Theii colleagues in other 
districts throughout the nation share this quest. The encouraging 
nationwide trends in reducing the achievement gap are due in large 
measure to such efforts. 
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The questions that continuously emerge as we strive to develop 
policies and programs to improve the academic and social 
performance of all our students and to regain the confidence and 
support of ou. publics include: 

• How to improve student motivation and behavior and thus 
reduce such crippling and costly problems as dropouts, 
absenteeism, violence, and vandalism? 

• How to better assess and improve the performance of our 
students and the effectiveness and success of our programs? 

• How to better identify and respond to the needs of special 
student populations; e.g., the disadvantaged, the 
handicapped, bilingual and multicultural, gifted and 
talented? 

• How to better select for and support improved staff 
competence in helping students gain academic, instructional, 
social, and personal skills? 

• How to better gain and equitably utilize funding for our 
systems that is adequate ind secure? 

• How u improve communication with and involvement of our 
parents and community? 

Answers to these pressing problems of education may be 
discovered if *ve continue to work toward creating: 

• Accurate, accessible, and timely data on the nature and 
extent of the needs of our students; 

• Valid, reliable, and efficient measures of the progress our 
students are making so that we can identify the programs 
and practices that help them overcome the impediments to 
success in school; and 

• Field-based as well as fundamental research on new teacher 
and technology -based approaches to enhancing the 
productivity of instruction and schooling in responding to our 
students 1 needs. 




The Portland, Oregon school district is one of those which has 
taken the data-based research and evaluation approach to defining 
and seeking answers to the perplexing educational, social, moral and 
political problems of differences in average levels of student 
achievement among student groups of differing ethnic backgrounds* 

The Portland Experience 

It was to help us answer questions such as the above that led 
the Portland, Oregon school district to develop an innovative testing 
system and to begin analyzing and reporting student achievement 
data as well as other student performance and status data by ethnic 
group* 

Testing and Test Results Reporting Innovations 

A decade and a half ago, a small group of researchers and test 
developers in the Portland, Oregon School District realized the 
potential of Item Response Theory for developing tools for better 
educational measurement. 

The purposes we wanted our citywide testing programs to 
support included equitable, effective, efficient, valid, and reliable: 

• Grouping and placing students 

• Targeting instruction on individual student learning needs 

• Evaluating student progress over time 

• Identifying neglected areas of the aligned curriculum and 
evaluating and improving programs and services at the 
student, classroom, grade within school, and grade within 
district levels 

• Providing accountability to the school board and the 
community 

In order to meet these needs, we needed an educational 
measurement system that would answer the following questions: 

• Is the current rate of gain of this student, class, grade, or 
program satisfactory compared to his/her/its age, grade, 
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program mates and the previous pattern of gains observed? 

• What are the current strengths and weaknesses (in terms of 
goal areas needing further diagnosis and possible work) of 
this student, class, grade, school, or program, and how have 
they changed over time? 

• Is the level at which this student, class, grade, or program 
currently performing satisfactory compared to his/her/its 
age, grade, or program mates and the previous pattern of 
levels observed? 

We could not find any available measurement program that 
would help us answer these questions and meet these needs 
adequately, and so we set out to build one ourselves. 

There followed a period of extensive collaborative research and 
development, much of it within the framework of the Northwest 
Evaluation Association, which was created to foster regional 
cooperation in and mutual benefit from this effort The result today 
is a system of three comprehensive basic skills Rasch calibrated item 
banks in Reading, Mathematics, and Language Usage. The constantly 
growing item banks in Reading and Language Usage each have over 
2,000 field-tested, calibrated items linked to a common, continuous 
curriculum scale for each subject. The Mathematics item t*ank now 
has over 3,000 such items. State and local school systems including 
Portland have been using these item banks since 1977 to construct 
effective, efficient survey achievement tests, competency tests, and 
other instruments that combine the best qualities of criterion- 
referenced and norm-referenced measurement. These excellent 
measurement systems have been the cornerstones of state, district 
and school renewal efforts that anticipated "A Nation at Risk 11 
(Gardner, 1983) by at least five years. The ongoing collaboration is 
now resulting in similar item banks in Science and Direct Writing, 
and yet another in Social Studies is on the drawing board. 

Some of the characteristics of the testing and test reporting 
system developed by the Portland School District which improve 
school and classroom effectiveness include: 

• Matches the local curriculum. 

• Emphasizes student gain over time (rather than just level of 
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performance at the current time) 



* Gives every student a challenging testing experience at which 
he or she will succeed (functioning level testing). 

* Reports goal areas in which students may need help as well 
as overall performance in each subject. 

* Monitors student progress toward mastery of graduation 
competency requirements starting with the beginning of 
third grade. 

* Invites parent involvement with teachers in planning to help 
students learn better. 

Two additional computer-based testing and test reporting 
innovations on which we are working are: 

1* Offering district-wide a school-based microprocessor test 
reporting system* Over the past two years, a group of seven 
principals, along with the data processing department and the 
research and evaluation department, have worked to develop 
this computer system for local building controlled reporting and 
analysis of test data. The pilot system began with four goals in 
mind. We were interested in finding a program that would run 
on building microcomputers that would accomplish the following: 

a. Provide a complete individual student test history to 
building staff immediately on request. 

b, Produce test reports by instructional group. 

c* Provide analyses of longitudinal student group data when 

and as the building needed them, and 
d. Improve the turnaround time of test reports. 

We now have a program which gives local buildings the ability 
to meet these four goals and we have offered it to all schools in the 
district, 

2. Creating a school-based computer adaptive testing system which 
allows building personnel to continuously monitor the progress 
of students as they advance through the basic skills curriculum. 
This system involves putting a sufficient bank of field-tested 
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and calibrated items inside a computer along with the requisite 
software to build a unique, individualized test for each student 
at the time when building staff feel it is needed. 

The advantages of this system include: 

• Increased measurement accuracy 

• Increased testing flexibility 

• Improved use of testing as an integral part of the 
instructional process 

• Enhanced test security 

• Decreased testing time 

• Increased ability to measure high-level educational goals 
such as problem solving 

• Immediate feedback of results 

Our research and evaluation department has mounted the pilot 
CAT program in cooperation with the information services 
department, directors of instruction, and principals. The purpose of 
this pilot is to gain the information necessary to design a cost- 
effective CAT system that will serve the future testing needs of all 
our students and our schools. 

Tested theories of effective education now reveal what we 
must do to improve schools and to help every child learn as much as 
he or she can as effectively and efficiently as possible. They indicate 
that our educational leadership must support the development of 
learning environments in which the following sorts of things happen 
for each of our students: L 

• His or her current, most pressing learning needs within a 
well planned curriculum, must be identified. 

• The student must be helped to set clear, relevant, attainable 
learning objectives to meet those needs. 

• He or she must be expected to succeed in attaining the 
learning objectives and must want to learn them. 

• The student must receive individualized instruction directly 
related to the learning objectives designed to meet his or her 
current learning needs. 
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* The learner must use the time allocated for instruction to 
work intently and seriously on the task of learning. 

* The student must know when he or she has succeeded and 
when not, and must experience a reinforcing sense of 
accomplishment and achievement as a result of knowledge of 
success. 

* The learner must receive and return a sense of caring, 
personal concern, interest, respect, and commitment which 
provides the psychological support necessary to want to 
learn and to work to learn, and finally 

* The student must receive and accept parental and 
community support and encouragement for success in 
learning « 

The main barrier to our putting such models of effective 
instruction and education into practice up until now has been the 
lack of accurate data and information about: 

* Each student's individual learning-needs. 

* What learning activities and experiences are matched to 
diagnosed student needs anu to established learning 
objectives and how to help the student engage in such 
tailored instruction in a timely fashion. 

* When the student has mastered the objectives and is ready 
to move on, 

* The degree of overall success of staff and programs in 
promoting student learning, and 

* What is and is not working to help students learn. 

We are now, however, at long last beginning to evolve the 
comprehensive assessment and information systems needed in order 
to create the more effective, equitable, and efficient education 
systems required for real and meaningful educational reform and 
even reinvention of schooling, 
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Guiding Values, Principles, and Goals of the Portland Effort 



The Portland initiative in using assessment data to improve 
school effectiveness, especially for under-achieving minority 
students, is founded upon the following values, principles, and goals: 

* All students can learn* 

We categorically reject the suggestions in some quarters that 
the observed differences in the achievement levels of students of 
various backgrounds are intrinsic. Instead we believe that the 
barriers to high levels of achievement by all students are 
surmountable and that research-based solutions can be used to 
create schools in which all students achieve at levels which will help 
them be productive members of society while meeting their personal 
goals, 

* Helping all students learn up to their maximum 
potential will require complex solutions to complex 
problems* There are no simple solutions or panaceas* 

We recognize that the existing research on obstacles to student 
achievement indicates that the underlying problems are: 

often long-term - beginning as early as the prenatal 
environment; and both complex and pervasive * with elements 
being found in the home, community, peer, and school 
environments and cultures. 

We have resisted the temptation to extrapolate from current 
trends and to project when the "achievement gap" will be closed. We 
are using the best data available at each level in the system to 
understand why seme students are not currently making the 
progress or performing at the levels we hope and we are developing 
and carrying out collaboratively developed plans to raise the 
achieveme n t of each individual student and each student group now 
lagging behind. 

* Teachers* principals* students* and parents are vital 
partners in this effort* 
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Disaggregated Reporting 

A key event in the maturing of this commitment was the 
decision of Superintendent Matthew Prophet and the Board of 
Education in 1935 to begin issuing an annual report entitled A 

Statistical Portrait of the MulticmturallMultiethmc Student Population in Portland Public 
Schools. 

This report represents only the "tip of the iceberg" of the 
district disaggregated data reporting system which extends to every 
level of the school district. It pulls together relevant district-wide 
analyses of data on culturally diverse children in Portland Public 
Schools to assist district decision makers in developing a general 
understanding of the status of these children In the district. The 
district-wide report presents findings in which the district bases 
planning of its efforts to address concerns regarding culturally 
diverse children. Similar data is reported and used at the program, 
building, and classroom levels* This promotes problem identification 
and solving at each key level in the system. 

At the district level the availability of disaggregated data helps 
t\e Superintendent, the Board of Education and the Administration to 
work with the community to muster resources and to formulate and 
monitor efforts to raise the achievement and education levels of all 
students, especially those lagging behind. Data-based research and 
evaluation on the causes of and solutions to problems causing lack of 
educational progress become possible at every level. Program, 
directors and building managers are helped to monitor and improve 
their units' efforts to bring about improvements. Teachers and 
others directly involved in instruction can evaluate and improve the 
effectiveness of their efforts to help all student groups and 
individual students progress through the curriculum. It is reasons 
such as these that have led Portland to take research, evaluation, and 
assessment-based approaches to understand and respond to the 
needs of under-achieving minority students. 

This district-wide report of data disaggregated by ethnic group 
was undertaken in collaboration with the district's Desegregation 
Monitoring Advisory Committee, a consortium of representatives of 
community groups concerned about equitable and excellent 
education and for education that is truly multicultural. The District's 
Management Information Systems group coordinated the d^ta 
collection, analyses and reporting effort. The assistance of the 




Northwest Regional Laboratory was obtained in designing and 
developing the initial report. 



The Portland Findings 

The Portland School District registers its students as American 
Indian, White, Black, Asian or Hispanic, For the 1986-87 school year, 
the student enrollment was 73.0 percent White, 153 percent Black, 
7.5 percent Asian, 2A percent Hispanic and 2.1 percent American 
Indian. The district has enjoyed a high level of desegregation and 
has never been under a court order to desegregate. It does have a 
voluntary desegregation/integration plan focused on improving 
student achievement and a Desegregation Monitoring Advisory 
Committee composed of representatives of community groups having 
a stake in educational equity and excellence. 

In developing out district-wide Statistical Portrait of the Multicultural/Multiethnic 
Siudent Population in the Portland Public Schools to symbolize and carry out our 
commitment to collaborative problem identification and solving with 
our community, we used a variety of sources to identify areas of 
district-wide concern regarding culturally diverse children. These 
included concerns expressed by the lay public and local school 
personnel. In addition we conducted an extensive review of the 
current literature and media stories to gain a broader perspective on 
concern: f local relevance. Through this process, we have identified 
the following five general areas for analysis: 

• Student Achievement. At what levels do culturally diverse 
children in the district demonstrate achievement? 

• Program Access. To what extent do culturally diverse 
children participate in district programs? 

• Multicultural Curriculum. In what ways does the district's 
curriculum address appreciation and knowledge of one's own 
culture or the cultures of others? 

• Teaching Personnel. To what extent does the district's 
teaching staff reflect the cultural diversity of its students? 

• Policy Representation. Do groups which formulate district 
policies reflect the cultural diversity of the students? 
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The report itself does not attempt to deal with the causes of the 
conditions it profiles. Instead, it presents an honest and accurate 
statistical portrait of cultuiilly diverse children in the district on 
selected variables for which data are available by ethnic group. The 
problem identification and solution process is inspired and guided by 
the data in the profile, A summary of findings follows to convey the 
nature and the power of the data presented. This data is revealed in 
far greater detail in the district's profile report and is articulated 
further at program, building and classroom levels where it is used as 
an integral part of school and classroom improvement, evaluation 
and planning. 

Student Achievement. 

We looked at five student achievement indicators: test scores, 
school grades, absences, dropouts and suspensions. In general, 
results from the elementary Portland Achievement Levels Tests 
showed that White students had a higher level of achievement test 
scores than other ethnic groups in both reading and mathematics. 
Minority groups, particularly Blacks, showed lower levels of scores. 
Longitudinal data obtained from the 1350-81 through 1985-86 
school years, however, provided some evidence that while the 
average levels of minority student scores were lower than the 
average levels of White student scores, the differences were 
generally becoming smaller over the years. Minority students have 
been making, in many cases, greater gains within a school year than 
White students. The result has been a convergence upon and in 
many cases surpassing of the national average as well as a trend 
toward closing of the "achievement gap* in many instances (see 
Figure 2). 
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FIGURE 2 



GRAPHIC PRESENTATION OF THE PERCENT OF PORTLAND 
PUBLIC SCHOOLS ELEMENTARY STUDENTS ABOVE NATIONAL 
TEST PUBLISHERS NORMS. 
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Approximately 7.6 percent * f high school seniors were 
ineligible to receive a standard high school diploma this year due to 
their inability to pass the ("strict's basic skills Graduation Standards 
Tests (GST), This was up from 5.8 percent in 1984-85 and 6.0 
percent in 1983-84, A generally disproportionate percentage of the 
students not passing the district's Graduation Standards Test have 
been minority students. This disproportion has been, however, 
declining. It has occurred primarily among those Asian students who 
were recent Southeast Asian refugees, In 1983-84, 52*3 percent of 
the students not passing the GST were Asian; in 1984-85, this 
dropped to 50.2 percent; and in 1985-86 the percentage dropped 
substantially to 35 5 percent. The percentage of Black students not 
passing the GST has also decreased over the past three years. In 
1983-84, 22.8 percent were Black; in 1984-85, the percentage had 
dropped to 18.6; and 1985-86,. 1 7.3 were Black, The number of 
Hispanic and of American Indian students not passing the Graduation 
Standards Test has been consistently low over the past three years. 

Asian and White students received the highest percentage 
(34*0 and 25*2 respectively) of A's a^id the lowest percentage (5,2 
and 10,2 respectively) of Fs in their schoolwork. In proportion, 
Blacks received the smallest percentage of A's (13*3 percent) and 
American Indians had the largest percentage of Fs (20,5 percent). 

Asian students have the highest GPA and averaged a total GPA 
of 2*80 (A B-X White students had the second highest GPA and 
averaged 2*43 (A C+)* Hispanic, American Indian, and Black students 
averaged a total GPA of about 2,0 (A C)* 

Asian students showed the smallest number of class absences, 
averaging 3.5 absences in ttz fir t of the day during the second 
quarter of 1986-87, American Ir K: Blacks and Hispanics had 
above average class absences dunn& ,ae sa ? period of time. White 
students averaged 5.2 absences, the same ;*> the district average- 
There has been a general decrease in the rates in which 
students have dropped out of school in any one school year, Asian 
students showed the lowest dropout rate, averaging 5,1 percent for 
1985-86, American Pdians had higher dropout rates, averaging 
13,0 percent for the same year, Dropout rates for White and 
Hispanic students were close to the district average of 7.7 percent 
Black students* dropout r?tes were well below the district's, 
averaging 6,0 percent for the 1985-86 school year. 
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The percentage of high school graduates differed substantially 
among the ethnic groups. Hispanic, White, and Black seniors 
demonstrated higher than average rates of graduation (96, 90, 89 
percent, respectively). American Indian and Asian seniors had a 
slightly lower than average rate of high school graduation. 

Suspension rates have general 1 ;- declined over the past several 
years for most ethnic groups. Black students, however, still showed a 
relatively high rate, averaging 5.9 percent for 1985-36 as compared 
with the district average of 3.7 percent. Asian students had the 
lowest rate, averaging only 1.2 percent for the same year. Hispanic 
and White students showed suspension rates lower than the district 
average. American Indian students' suspension rates were up during 
the 1985-86 school year, but numbers of American Indian students 
suspended are so small that percentages are generally not consistent 
from year to year. 

Program Access. 

In this analysis, we examined six program areas: talented and 
gifted programs, high school magnet programs, special education 
programs, ESL/bilingual programs, Chapter I programs and Project 
SEED. 

Asian and White students showed the highest participation in 
TAG. Their rates, 8.5 and 8.1 percent respectively in 1986-87, were 
more than twice as high as those of other ethnic groups. American 
Indian, Black and Hispanic students had a participation rate of about 
3 percent. 

Black students have the highest participation in the magnet 
programs, averaging 11.2 percent in 1986-87. Hispanic students also 
showed an a!^ove average rate of 9.3 percent for taat year. 
Participation rates of American Indian, White, and Asian students 
were slightly beW the district average of 8.6 percent. 

There was a slight increase in special education enrollment 
through 1984-85 and since then there has been a slight decline. In 
1986-87, the percentages of American Indian (12.3 percent) and 
Black (12.8 percent) students receiving special education services 
were higher than those of the other ethnic groups. Asian students, 
on the other hand, had the lowest rate of participation, averaging 2.2 
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percent. Enrollment of White and Hispanic studeius was slightly 
below the district average of 8*5 percent* 

For the 1986-87 school year, Black participation in Chapter 1 
(22*6 percent for reading and 17*0 percent for math) was more than 
twice as high as the district averages of 10* 1 and 6+7 percent. 
Participation rates for Asian, American Indian and Hispanic students 
were higher than the district average. White students had the 
smallest enrollment in Chapter I, averaging 7*0 percent for reading 
and 4*2 percent for math. 

In proportion, more Black students were enrolled in Project 
SEED than any other ethnic group* Their pamcipation rate of 73 
percent for 1986-87 was more than four times as high as the district 
average of 1*6* Asian, White, and American Indian students had the 
lowest rate, averaging approximately 0*5 percent* Hispanic students 
were close to the district average at 1*7 percent* 

Multicultural Curriculum. 

The district appears to have made a concerted effort to 
strengthen its multicultural curriculum* The Curriculum Depart* + ent 
has identified six major geocultural groups which have contributed to 
the American cultures* It has used this categorization to organize its 
multicultural curriculum. Each of the eight core curriculum areas has 
goals and objectives for teaching content that is multicultural with 
specific reference to the six major geocuhural groups* The 
department has developed a cross-referenced grid linking existing 
planning documents, instructional materials, and key cultural 
concepts with the major ethnic groups* 

In addition, the Educational Media Department, through the 
district's Professional Library, its Central Audiovisual Library, and 
individual school media centers, provides resources for both teachers 
and students in multicultural education. And the ESL/Bilingual 
program provides new student orientation, appreciation and support 
for home language and culture, and awareness of American's 
multicultural society; it has developed cun ulum materials for 
students from diverse linguistic and cultural backgrounds* 

Teaching Personnel* 

At each individual school, the ethnic distribution of teachers 
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somewhat corresponds with th&t of its students. Schools with an 
above average percentage of one group of students often have aa 
above average percentage of teachers of the same group. Looking at 
the district as a whole, however, one finds a consistent pattern of 
discrepancies. Except for White students, teacher percentages are 
generally less than one-half of the respective student percentages. 
There is a discernible pattern that teachers of groups other than 
Whites are under-represented in rhe sealing staff. The 
discrepancies between teacher and student ethnic distribution are 
most pronounced at the high school level. In addition, for the past 
two years nearly 90% of ail newly hired teachers have been White in 
spite of intensified efforts to reciui; minority candidates. 

Policy Representation- 

We have reviewed the ethnic composition of district central 
administration, school principals, Citizens Advisory Committees 
(CACs) and the Desegregation Monitoring Advisory Committee 
(DMAC), The data indicate that three groups (American Indian, 
Asian and Hispanic) were consistently under-represented in such 
policy groups. Blacks were over-represented in central 
administration and DMAC but under-represented in CACs, 

In each area of review, the district has developed plans to 
improve the education of its culturally diverse children. For 
example, the district will expand the levels testing to include more 
subject areas (e*g», science and social studies) and grade levels. It 
will continue to review test items and testing procedures for 
potential bias and develop more sophisticated ways of analyzing dau? 
on student growth. In addition, the district is working on data 
systems which will allow for longitudinal tracking and analysis of 
student data on school grades* absences, dropouts and suspensions. 

To improve program functions and outcomes, the district is 
reviewing and analyzing ways in which it has operated its special 
programs, including TAG, magnet, special education, ESL/bilingual, 
Chapter I, and Projects SEED and MESA, 

The Curriculum Department has developed a series of plans to 
revise and expand its multicultural resources and materials as well 
as in-service training for teachers. 

The Personnel Department is working to find ways of attracting 
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and retaining minority teachers. 



The Superintendent is working with several community grou; s 
in an effort to encourage and support involvement of our culturally 
diverse citizens in school district affairs. 

Syntheses 

Table 1 presents the difference between the averages for each 
ethnic group and the district averages on the variables studied. For 
student achievement data, ethnic group averages are compared with 
averages for all students For program access data, participation 
rates of each ethnic group arc compared with the district 
participation rates. 

Comparing ethnic group and district averages provides a 
picture of above district achievement, below district achievement, 
over-representation, and under-representation. In Table 1 , a plus 
sign indicates that the ethnic group is higher than the district 
average; a minus sign indicates the group is below the district 
a vera ce. 



TABLE 1 

SUMMfAY INFORMATION ON CULTURALLY LIVEKSE STUDENT GROUPS 

1986-87 



American 

Variable Indian White Black Asian His panic 

8th Grade Reading: 

Level - 4 

Gain + + + + 

8th Grade Math: 

Level - * * - + 

Gain +■ - - + + 



Passing Graduation 

Standards Tests * * 
School Grades: 

A's + - * 

F's ♦ - ♦ - + 

Class Absences ♦ - ♦ * 

Graduation Ratio - * - * * 
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TABLE 1 (CONT.) 
SUMMARY INTORMATICN ON CULTURALLY DIVERSE STUDENT GROUPS 

1986-87 



Variable 


American 
Indian 


White 


Black 


Asian 


Hispanic 


Dropout Rates 
Suspension Rates 


+ 
+ 


+ 


+ 
+ 


- 


+ 


Program Access: 
TAG 
Magnet 

Special Ed. (84-A5) 

Chapter I 

SEED 


+ 
+ 


+ 


+ 
+ 
+ 


+ 


+ 
+ 


Teaching Personnel: 
Elementary 

Middle 
High 




+ 
+ 
+ 








Policy Representation: 
Administration 
Principal 
CAC 
DMAC 




+ 
+ 


+ 







Note: Only the niost recent data are included in summary. 



+ = Above district average 
- * Below district average 
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As the district- wide synthesis in Table 1 indicates, there are 
still significant discrepancies in indicators of student performance 
among student ethnic groups in the Portland School District. 
Although, as noteo earlier, some progress has been made in reducing 
lag in achievement test score level by greater gains by minority 
student groups, the most encouraging progress is indicated by other 
indicators such as reduced dropout and suspension rates. The key 
philosophical and procedural difference is that we are using such 
data at all appropriate levels throughout the system to plan, 
implement and monitor the success of efforts to reduce 
discrepancies. One key early benefit of this open data sharing has 
been an improvement in the climate of community relations. 

Some of the major causes of this improved success of our 
students and our school system are the implementation of carefully 
screened and piloted specific programs and implementation of the 
general effective schools philosophy including: 

♦ Improved classroom management aimed at helping students 
become more motivated and more responsible for their 
behavior and their learning. This results in fewer 
interruptions for teachers and students and more time on the 
task of learning. 

♦ Clear and high goals for achievement in the basics and 
beyond. Our teachers know what they want students to 
learn and students understand what is expected of them. 

♦ Well selected and appropriately challenging curriculum 
materials, instructional systems and learning tasks. 

♦ Instruction designed to challenge every student to succeed 
and targeted on his or her current diagnosed learning needs. 
Such individualized and personalized instruction brings out 
the best in each learner. 

♦ Effective and efficient special prograi. , aimed at helping 
teachers meet the special needs of students. 

♦ Partnerships with parents and with community groups in 
supporting student learning. 

♦ Prompt, accurate and frequent feedback to students, parents, 
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teachers, support staff, program personnel, District managers, 
and policy makers on the success of their efforts at learning 
and at fostering learning. 

• Systematic recognition, rewards and incentives for excellence 
in the level of achievement and progress in learning. 

Conclusion 

A variety of theories and hypotheses have been advanced to 
explain the observed differences in performance by various groups 
of students. Environmental theories held sway in the I950*s and 
1960's, with an emphasis on family and school environments. The^e 
theories gave rise to the school desegregation and compensatory 
education movements. In the late sixties theories of cultural 
difference were developed. These paved the way for the 
multiethnic/multicultural movement of the 1970's and 1980's. A 
parallel development during the same period has been the 
emergence of the mastery learning and effective schools' movements 
with their emphasis on the ability of all students to succeed given 
clear and high expectations, emphasis on the basics, a safe and 
orderly learning environment, instructional leadership, and regular 
monitoring and reporting of student and program success. (Edmonds, 
1979) 

The fact remains that there are considerable differences in the 
observed average levels of academic achievement of students of 
different ethnic groups as early as first grade or even kindergarten. 
While research has established that much of the observed 
differences are correlated with socio-economic variables and while 
recent trends point to above average gains for previously low 
achieving student groups there is still much to be dene to understand 
and deal with differences in student and system achievement and 
performance levels and with the subtle and complex educational and 
social issues that underlie them. 

The steps being taken by Portland give hope of data based and 
measurement supported understanding and progress. We are 
developing and sharing with our staff, students and communities 
honest, accurate information on the performance of student groups 
and we are using that information to plan change and to monitor 
student and program success. As much as we would all like things to 
be perfect, as educational managers and as educators, we are 
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responsible for knowing what is the case, in working to make things 
as good as we can, and for knowing when, why and how we have 
succeeded and failed and then trying to do better next time. 



The staff of the Portland Public Schools are proud that we have 
risen and are continuing to rise to the difficult challenge posed by 
student performance differences between student ethnic groups. In 
general our efforts have been well received by our community which 
welcomes accurate, timely and freely available information on how 
their children are doing and who value the new opportunities we are 
creating to work together to improve education for all students. 
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Footnote 

1. In a recent report (June, 1987) Lyle Jones questions whether 
young black children in impoverished rural and urban areas are 
participating in the overall closing of the Reading NAEP Achievement 
Gap. 
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THE ROLE OF TESTING AND EVALUATION 



The Need to Assess Multiple Crucial Components 
in Evaluating Programs 

Daniel Levine 

University of Missouri 

My goal in this brief paper will be to provide several 
illustrations of the importance of attending to multiple crucial 
components of effective programs and determining how they may fit 
together to improve instruction for disadvantaged minority students. 
Such attention is critical in the identification, design, implementation, 
and evaluation of effective programs. 

One useful recent example of the general problems that arise in 
identifying and assessing crucial components has been provided in a 
"best-evidence" analysis of mastery learning research carried out by 
Bob Slavin of Jon Hopkins University. Zeroing in on technically- 
competent studies that compared mastery learning with traditional 
instructional sequences, Bob concluded that mastery learning 
generally has not produced large, generalizable achievement gains in 
controlled experimental implementations. 

This important finding points (in my mind) to the likelihood 
that mastery learning along will not help low achieving students in 
poverty schools or elsewhere unless it is implemented in conjunction 
with other key changes such as effective curriculum alignment and 
reorganization, if necessary, in grouping arrangements. In my 
experience, some schools that have attended to such imperatives as 
part of a unified approach to improvement have been successful in 
substantially raising student achievement through mastery learning. 
In short, mastery learning may be a crucial component in some 
successful efforts to improve achievement, but it alone is not 
sufficient to assure program effectiveness. Rather, a multiplicity of 
crucial components is required. 

Rather than negating the potential utility of mastery learning, 
Bob's results can be viewed as helping to call attention to the 
concomitant changes that must be made if master/ learning is to be a 
useful component in a school improvement project. Unfortunately, 
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his results likely will be used by some readers to support the 
conclusion that mastery learning cannot be helpful in improving the 
achievement of disadvantaged students. 

One useful way to articulate the larger issue of multiple crucial 
components is to pose and try to answer a few questions as follow: 

Q If mastery learning or any single treatment by itself, does not 
improve students' achievement, does not mean it is not 
potentially potent in helping to improve achievement? 

A: No* Successful mastery learning at poverty schools requires, 
among other things, implementation in conjunction with 
unusually effective organizational arrangements for low 
achievers* (One possibility involves very small classes for 
students functioning very poorly.) In addition, by definition 
mastery learning requires more time to provide corrective 
instruction for low achievers* If mastery learning when 
properly implemented in conjunction with appropriate changes 
in organization and scheduling of instruction yields large 
achievement gains, then mastery learning can be an important 
approach for improving achievement* An exact analogy would 
be a medical treatment in which exercise and medication 
together, but neither separately, reduced subsequent incidence 
of heart attacks* 

Q What then can we learn about the effects of potentially 

important instructional changes when we vary conditions in 
order to test them in isolation? 

A: Only whether they are effective in isolation, not whether they 
can be combined with other changes to produce improvement* 
Since instruction takes place in the complex setting of schools 
and classrooms, few if any innovations are likely to produce 
sustained and substantial improvement unless part of a larger 
effort to impact the larger setting. It is well established, for 
example, that substantial staff development, together with 
motivation to participate in it, is a prerequisite for successfully 
implementing a serious innovation. Given this 
interdependence one should not reject an innovation after 
assessing Us effects in settings with inadequate staff 
development or insufficient incentives and support* 
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Q Is mastery learning the only instructional sequence available 
for improving the achievement of disadvantaged students? Is 
there a single best approach? 

A: Obviously not* In fact, most of the unusually successful 

poverty schools I have seen or learned about have not used 
mastery learning as defined by James Block, Benjamin Bloom, 
S. Alan Cohen, Thomas Guskey ? or other leaders in this field. In 
addition, I have not been able to find inner city senior high 
schools (or hardly any other high schools, for that matter) 
which have introduced mastery learning successfully on a 
school-wide basis* On the other hand, successful inner city 
schools of necessity do use some approach to "mastery-type" 
learning, i.e. their faculty go to whatever lengths are necessary 
to make sure that nearly all their students m:±z progress in 
mastering agreed-upon learning objectives. Within this 
context, when mastery learning as defined by Bloom and others 
is implemented well, it does offer some particular advantages 
in terms of focusing instruction more effectively on the 
learning problems of initial low achievers* 

A related problem occurs in situations in which the set of 
changes or variables one is assessin* through research does not 
include those that actually were most important in bringing about 
improvement* One example of this occurred in the series c/ sub- 
studies which researchers at the old U.S. Office of Education 
conducted using data from the EEO study directed by James Coleman. 
Because the questionnaire administered at schools participating in 
the study did not include good items dealing with the leadership of 
the principal there was no possibility that this variable could show 
up well in the sub-study that contrasted unusually effective and 
ineffective schools. . 

Another possible example along these lines may be present in 
some of the publications prepared by personnel in the San Diego 
Unified " ^ District* San Diego has carried out probably the most 
success ogram in the United States for improving the 
perform * of students attending concentrated poverty schools -- 
the Achievement Goals Program (AGP)* The descriptions I have seen 
of the AGP unusually cite four main components: increased ti me on 
task: direct instruction: improved classroom management: and 
mastery learning . 
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However, there are reasons to believe that an equally or even 
more important intervention involved a radical curriculum change 
which removed basa 1 readers from classrooms so that teachers no 
longer could proce* 1 page-by-page at the pace of the slowest 
student (This intervention is only hinted at in some descriptions of 
the four components.) If this intervention indeed was critical, as 
there is reason to believe it was implementation of an AGP-like 
program elsewhere in the absence of radical curriculum alignment 
could result in non-transportability along with severe 
disappointment and discrediting of the four useful components 
identified as part of the program. 

Among the implications of the preceding discussion are that 
program evaluation generally requires some attention to 
implementation analysis, whether formally labelled in this way or 
not, and that evaluators should have substantial knowledge of and 
familiarity with the schools at which they assess programs. 

On th. first point, it is apparent that evaluators must 
understand what schools actually are doing to implement program 
components and to overcome obstacles to effective implementation, 
if components crucial to success are to be identified for the analysis. 

Regarding the need for evaluators to acquire in-depth 
knowledge of how programs actually are being implemented and the 
actions teachers and administrators must take to assure their 
success, evaluators can spend time in classrooms themselves or 
obtain the opinions and perceptions of others familiar with 
participating schools, or combine these two approaches. 

Of course, spending time in schools and obtaining information 
from knowledgeable observes do not guarantee that crucial 
components will be clearly identified, but failure to engage in these 
aspects ci data collection will multiply the likelihood that program 
assessments will omit key considerations witici. ultimately determine 
success or failure among initial participants and subsequently at sites 
engaged in replicating promising innovations. 
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THE ROLE OF TESTING AND EVALUATION 



Some Thoughts on How Testing and Evaluation Can Improve 
Educational Opportunities for Underachieving 

Minorities 

Ernesto M Bernal, Ph*D, 
Northern Arizona University 

A twenty minute presentation is not very long, yet the 
opportunity to address such a diverse and august group of educators 
prompts me to focus my time on some thoughts which I hope will 
m?ke you somewhat uncomfortable, in some cases for their 
specuiativeness and in other instances for their departure from the 
current Zeitgeist. I will address both testing and evaluation, or 
rather, some new departures in both fields which I believe are 
definable in the light of what is needed, or what has not yet been 
accomplished. 

Evalua tion 

I am gratified that the number of journals on evaluation has 
increased but concerned that the number of journal articles dealing 
with urb?m + minority, and compensatory education has not. A cynic, 
I suppose, could begin by asking, ''Given that much of the financial 
impetus for the development of program evaluation as a field came 
from compensatory education and other government programs 
(Daniels & O'Neil, 1979), whut has evaluation done to improve the lot 
of the underserved, the ill-served, non-dominant groups in American 
society?" One of CRESST^ guiding premises is the belief that testing 
and evaluation are important tools for promoting ^Jucational equity 
Perhaps this conference is a partial repayment, 

But to move ahead, I believe it is necessary for us to 
distinguish between a moderately lOutinized, operational program 
evaluation and one which seeks to influence policy at levels which 
could affect a significant sector of educational practice, not just a 
parochial interest. 

We need also to recognise that certain programs are likely to 
generate confusion and ill will when they deal with an already 
politically sensitive area (e.g., bilingual education), require extensive 
systemic change (e.g., Experimental Schools Program) (Lenning, 
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1977), or combine one or both of these elements with personal pride 
ft finances (as in the case of Career Ladders). It is wise to note that 
tor some proponents and opponents of controversial programs such 
as these, no amount of data or evidence will dissuade them. What 
we should attempt is to gain a balanced picture when analyzing 
large-scale, controversial programs. One could, for example, let out 
three small contracts instead of one large one. One proponent group 
and one contrary group would be in charge of analyzing the data 
which yet a third, ostensibly more neutral, group would collect to 
satisfy the analysts' designs. Comparing the results (reported in a 
uniform, juxtaposed format) might help us all to evaluate how well 
evaluation can handle controversy (see Duckett u ah, 1982) and see 
just how Suchman's old (1967) pitfalls of evaluation ("eyewash," 
"whitewash," "submarine,", etc.) apply today. 

It is a testimony to our lack of imagination that comp-ed 
programs look so compensatory. (Like Levin, I feel that comp-ed 
should be an enriching, alternative mode of delivery.) And it is a 
similar problem for us to continue to look at national norms and 
"regular progress" as the touchstones for many minority programs. 
We need to think of how educationally disadvantaged minority 
children would do if they were placed in the re gular program, which 
after all does not want them (Bernal, 1984). (Why else were special 
programs created?) I am not sure how best to estimate these effects, 
but I am certain that this would be a more revealing comparison, 
both pedagogically and politically. 

Since minority students are culturally (i.e., behavior- 
ally) different, it is crucial that policy -and- program-evaluations deal 
with naturalistic settings (Wardrop, 1971), particularly the way the 
schools (representing the majority culture, values, and expectations) 
interact through their rules and representatives with the minority 
cultures, children, parents, and neighborhoods. I recall an old 
ethnographic study by Spradley (1971), never picked up in the 
education literature, which found that minority school children, 
rather that being "culturally deprived" were often culturally 
overwhelmed and concluded that to succeed in school minority 
students needed to possess a capacity not required of others, namely 
to become bicultural. This study illustrates how perceptions and 
protocol are critical intervening variables. Real events , not just 
official events, in terms of human transactions, must be documented 
(Charters & Jones 1973). 
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In a similar vein ? we need to stop fragmenting educational 
objectives without attempting a synthesis which transcends 
individual objectives (see Page & Stake, 1979), Many programs 
present an evaluator with goals and objectives which later 
experience proves to be excessive, undesirable, or positively 
misleading — B.S., in short. I believe that controversial, complex 
programs are not only not evaluatable in their first year, but also 
that the task of the evaluator of these programs is to complete ihe 
evaluation design after the project is underway, and that the first 
year or two of the evaluation should be as fluid and formatively 
dynamic as the project itself (see Weiss, 1973), 

Nor need we be naive to think that the goals and objectives 
provided us are the only ones we should measure. I suppose I am 
arguing against the disinterested evaluator role whenever policy 
issues are being investigated. The evaluator in a collaborative role, 
however, needs to have the courage to supply essential goals and to 
examine the data to see if they have been fulfilled. Among these are 
the long-term monitoring of achievement, placement, and retention 
of minority students to see if they have similar options for being 
'.tacked within the curriculum as White students do, over the long 
iun< (See Oakes, 19S7, for "indicators of equity,") 

While systems exist for objectively discovering which goals are 
real and for prioritizing the maze of objectives which complex 
programs present us (e.g. Borich, 1984), what is needed, 
additionally, is a good model for detecting how objectives interact 
with one another, particularly how implementation of certain 
program features — how the attainment of certain objectives — 
might impede or facilitate the realization of other objectives. For 
example, the ear'y reclassification and exit of limited-English- 
proficient (LEP) students from bilingual programs may impede ther 
long term success in school and necessitate further special services, 
such as Chapter I programs. (Such "exits" from transitional bilingual 
education to another compensatory program instead of to regular 
education are a sham, in my opinion.) The possibility that objectives 
interact with each other means that a project's effects and impacts 
may be greater or less than the sum totM of its individual objectives. 

Testing 

My first and most basic recommendation for psyc home trie ians 
is to move directly to de-bias tests of intelligence, aptitude, and 
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achievement, even if you do not believe them to be biased! This 
effort would in any case be a more creative and engaging enterprise 
than defending current practices. New goal: to measure adequately 
as well as validly without compromising other features of a test like 
reliability and usability. 

I suppose that I sec recent developments in the bias issue as 
portending even greater conflict, more serious confrontations. On the 
one hand, such programs as "teacher-competency" and student- 
achievement (basic skills) testing, are expanding and generating new 
hundred" of thousands of dollar to certain test-making and test- 
scoring enterprises yearly. On the other side of the political 
equation, groups that oppose testing in one or more of its forms are 
joining forces with consumer advocates and even with avowedly 
political action organizations. The underlying socia! issue, however, 
has to do with the real costs of these conflicts in terms of human 
potential lost, not to mention the professional energies ;vhich will be 
dissipated in formal judicial hearings and legislative manipulations. 

Mercer (1979) has stated that psychologists generally are 
among the established American elites (in the sociological 
sense) and that they understandably perpetuate a 
psychometric belief system which provides a "scientific 
rationale for the continued ascendancy of politically dominant 
racial and cultural groups 0 (p. 112)* She further points out that 
there are a few psychologists who hold a counterideology, one 
which rejects "a definition of Intelligence* which is based 
entirely on an individual's knowledge of the Anglo core-culture 
and would include the language, skills, and knowledge needed 
to operate successfully in non-Anglo cultural settings" (Merce;, 
1979, p. 112), I wish to add that while many apologists for 
extant testing practices claim that measured mean differences 
between Whites and minorities are functions of differential 
educational opportunities, the testing industry has never really 
studied this particular question, a crucial hypothesis, really, 
which can only be answered by investigating several 
alternative possibilities at the same time. 

The new tescs which I envision might include a representative 
sampling of thinking, learning, and expressive styles and bicultural 
survival competencies (so that almost everyone does poorly on a few 
sections of the test, b, t not on the same ones). These tests should 
not be like the old "culture free 1 * tests, with an their attendant 

233 



74 



problems, including lack of validity, but should address themselves 
to a wide variety of abilities (see Flaugher, 1971) that new research 
would show are important to success if) life generally and in school 
as well. We must keep in mind that current predictor variables are 
not all that powerful (which occasionally gets tests into trouble even 
with White populations); hence a search for stronger and 
"alternative" cognitive skills and "noninte Elective" (Lenning et ah, 
1974) factors is in order — predictors we do not yet tap. We could, 
for example, develop tests based on studies of extreme groups who 
do not perform as our contingency tables would predict, then 
compare these groups to each other and to groups of more consonant 
individuals. The applicatic , of com^'ter-assisted testing may also 
be in order here, for the sake of both efficiency and humaneness, to 
reduce overall testing time and stress on such diverse sets of items. 

While tests allow us to do some very sophisticated analyses of 
individuals and groups, we have to adopt a more user-and 
consumer-friendly approach: 

1. We need tests of achievement/placement which yield not only 
reliable but also accurate scores without having first to subject 
minority kids to extensive test-taking skills training. In short, 
we need tests which are not so artificially constructed that they 
mask the true abilities and achievements of minority students. 
(See Bernal, 1986.) 

2* We need tests of ability/diagnosis which yield educationally 
and clinically meaningful profiles » 

3. We need tests which prognosuato success or failure in the long 
run, not just the short run, so that wt might better counsel 
both minority and majority students an1 prepare more 
appropriate interventions for those who may encounter only 
short-run difficulties. 

Finally and very importantly, we need to stop testing for the 
prestige of it or to gain political approbation from the public. 
Teacher competency testing, where a test score effectively becomes 
eirher the sole criterion or part of a multiple cutoff system of 
screening for admission to teacher education or for fortification, is an 
issue in point* The Pre-professional Skills Test (PPST) and the 
California Basic Educational Skills Test (CBEST) admittedly have Httk 
to do with actual professional outcomes such as classroom 
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effectiveness* Glass (1986), who concludes that very few people 
ultimately fail these tests (given that some must retake the 
examinations), misses an important point: Minorities fail these tests 
in disproportionately high rates and are not particularly successful 
on subsequent attempts, as the PPST data from Arizona indicate 
(Cropper & Nomura, 1987)* When we use tests to satisfy political 
agendas, tests which have little or no relationship to professional 
competencies, but which severely and disproportionately impact 
minorites, one must suspect that hidden political agendas may 
include the limitation of opportunities for minorities to enter the 
teaching profession while reassuring th? public that educational 
reform — and insistence on standards — is taking place. 

Conclusion 

Compensatory education has enjoyed some successes but 
perhaps not as many as it might have achieved had it embarked on a 
different course by offering a program more appropriately tailored 
to the needs of non-dominant ethnic schoolchildren. In too many 
cases, compensatory education has merely presented a slower, less 
interesting, less challenging version of regular education to these 
youngsters (see bernal, 1984), My own impression is that the results 
of these programs are ideologically disappointing although, 
realistically speaki.ig, quite good, given hew little innovation was 
invested in their design. 

Evaluation can make a real difference in the configuration and 
delivery of programs for underachieving minorities if it begins to 
document what is really going on, how these events are perceived by 
different actors, and how these perceptions compare. My experience 
convinces me that to a very great extent parents, children, 
administrators, and teachers do net perceive the same educational 
events in the same way, and that they might accomplish a lot more if 
they knew what each of the others was thinking, I suppose that I 
am arguing for more observation, more ethnographic monitoring, 
because I have encountered so many dissonant perceptions in my 
own studies. But because these require significant outlays of money, 
they may have to be reserved for major evaluative undertakings. 
What must be done in all cases, however, is to link these goings-on 
(programmatic features, events, variations) to student achievement, 
attendance, retention, and placement/tracking over long periods of 
time, since these criteria are closer to what educational equity is all 
about. 
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For the testing profession to make a contribution to minority 
education may require a major reorientation of both individuals and 
organizations, from defensive (albeit "scientific") posturing to 
creative problem -seeking and problem-solving. It would be far 
more socially beneficial and professionally challenging to discover 
the psychological sources of differential ethnic performance on both 
predictive and criterial measures than to continue to invest resources 
in the defense of traditional tests. 

If such research proves fruitful, if it were possible to sift the 
"real" differences from the culturally/arbitrarily imposed 
biases/problems in the instruments, then new tests could be devised 
which would alter not the psych ometrician's social role as gatekeeper 
(elite) but the demographic characteristics of the persons who would 
be most significantly affected by testing. 

By ensuring that both our instru ments and the criteria by 
which we judge their validity are unbiased, we could simultaneously 
assure ourselves, the public, and the school professionals whose 
programs we evaluate that our tests are not merely consequential 
but also germane. At a time when we spend such great efforts 
justifying current practice in test design and validation ;rnal, 
1986) to no one's deep satisfaction (except for true believers), is this 
not the time to try? 
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THE ROLE OF TESTING AND EVALUATION 



Results from Using The WICAT Learning 
Solution for Underachieving Minority Students 

James B< Olsen 
WICAT Systems 

Introduction 

This paper describes some very promising results from using 
the WICAT Learning Solution (LS) to produce significant learning 
gains for minority students. The Learning Solution involves the 
integrated use of the following; 

• a computerized school learning center, 

• comprehensive courseware curricula, 

• sophisticated learner management, 

• advanced assessment and testing programs, 

• hardware solutions designed for schools, 

• on-going training and support, and 

• complete maintenance service. 

The Learning Solution is currently implemented in 
approximately 350 locations nationwide. Representative school 
districts include: Garland District, TX; Pharr-San Juan-Alamo District, 
TX; Stillwater, OK; New York City, NY; Chicago, IL; Prince Georges 
County, MD; Broward County, FL; Indian River County, FL; Asuza 
District, CA; Hueneme District, CA; Oxnard District, CA; and Santa 
Barbara, CA. Each of the 350 school implementation sites has a 
minimum of 32 student learning stations (350 sites x 32 learning 
stations = 11,200 learning stations) which are used throughout the 
school day to provide comprehensive, computerized curriculum, 
testing, and instructional management capabilities. Seveial districts 
(Oarland, TX; Pharr-San Juan-Alaivo, TX, Indian River County, FL; 
Hueneme District, CA and Asuza District, CA) have implemented the 
Learning Solution in each school in the district. 
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Learning Solution Description 

The Learning Solution consists of the integration of a 
computerized school learning center, comprehensive courseware 
curricula, sophisticated learner management, advanced assessment 
and testing programs, hardware solutions designed for schools, on- 
going training and support, and completes maintenance services. The 
following sections describe the major features and capabilities of 
each of these integrated aspects of the Learning Solution. 

The School Learning Center. The Learning Solution is 
implemented in a school using a Learning Center configuration. The 
Learning Center includes minicomputer with a large hard disk 
storage capability which can support up to 32 student learning 
stations (standard school learning center) or up to 64 student 
learning stations (large school learning center). Alternative Learning 
Center configurations can provide individualized computerized 
instruction and assessment for up to 350 students daily (standard 
school learning center) or up to 700 students daily (large school 
learning center). 

The Learning Center provides lessons which are tailored to the 
individual learner Each lesson is automatically tailored to meet the 
needs of individual students. Interactive exercises encourage 
students to develop higher order thinking skills. Drill and practice 
lessons are also included. Teachers receive up-to-the minute student 
management reports which permit them to monitor student 
performance and progress and identify areas where students require 
individualized attention. 

The Learning Center allows each student to work on different 
courseware lessons all at the same time in the same lab. The system 
is also easy for teachers and students to learn and operate. Each 
student can take computerized achievement tests in the Learning 
Center and receive appropriate prescriptions to WICATs 
comprehensive curriculum. Student responses to the courseware and 
testing materials are monitored and teachers receive reports on 
individual performance as students proceed through the materials at 
their own pace. 

The Learning Center courseware uses an extensive graphics 
library to stimulate student interest and motivate students toward 
higher performance. Students see learning situations that are acted 
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out in vivid animation. Atl system configurations include high- 
quality, life-like audio capabilities which are educationally 
invaluable for beginning reading instruction, activity directions, 
language instruction, and feedback or reinforcement. 

Courseware Features 

The Learning Solution provides comprehensive K-12 grade 
courseware curricula for students in reading, mathematics, language 
arts, and speciality subjects. Currently, WICAT provides 15 year- 
long courseware curricula packages. These include: 

Mathematics: K-6 Mathematics, Middle School Math, Algebra I, 

Algebra II, and Geometry; 
Reading: Primary Reading, Reading Comprehension; 
Language Arts: Language Arts, Secondary Language Arts, and 

Writing; 

Speciality Subjects: Computer Literacy, En% *h as a Seconu 
Language, French, Chemistry, and 

High School Basis Skills: Reading, Math, and English, This 
represents u loial of approximately 2500 hours of computerized 
instruction. The comprehensive courseware has been developed to 
correlate to major state assessment objectives as well as to focus on 
teaching higher order thinking skills as well as basic skills. Yearly 
enhancements are made, as needed, for each courseware package. 
Brief examples are provided below of some of the typical courseware 
features. 

The WICAT Primary Reading Curriculum complements and 
enhances classroom instruction in beginning reading skills for 
students in kindergarten through grade 3, Through the use of voice- 
quality audio and powerful graphics, the curriculum teaches the 
following sample primary reading skills: letter identification and 
discrimination, initial consonant sounds, sight word identification, 
sound patterns, picture sentences, parag_ph comprehension, word 
identification through context, r M identifying word meanings in 
context. The curriculum is easy for students to use since most of the 
exercises require the use of only five basic keys* Several torms of 
Prompts and Helps are available to students. The Primary Reading 
curriculum consists of a total of 1010 interactive activities organized 
into 285 lessons. 
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The WICAT Reading Comprehension curriculum is appropriate 
for students grades 4-8 and includes 565 separate lessons, presented 
as newspapers articles or stories. The newspaper editions available 
to the student include a mix of siories on the student's reading grade 
level, one year above reading level, and one year below reading 
leveL From the newspaper edition list presented, the student selects 
a newspaper edition to work on and one of five newsstory articles 
available in the edition. The student must respond correctly to 80 
percent of the questions on three stories within one reading level 
before advancing to the next level. Various D rompts and Helps are 
available to students during the reading exercises. This unique 
curriculum teaches critical thinking skills and shows st'idenis how to 
apply these skills in a logical process to understand printed material. 

The curriculum teaches students to draw inferences and 
conclusions, and to make predictions from text; to provide 
justification for conclusions, inference predictions; to judge the 
validity of an argument based on stated criteria and evidence; 
determine the relations of parts of a passage to its total meaning; to 
interprete data presented in graph, chart, or tabular form; 2nd to 
identify appropriate summaries of text passages. 

The WICAT Middle School Math Curriculum is designed as an 
effective supplement to the standard mathematics curriculum for 
grades 5 through 8* The curriculum uses a variety of methods to 
teach mathematics concepts, to reinforce skills and to stimulate 
student interest. The Middle School Math Curriculum consists of 130 
lessons denized into the following five ,najor strands: 

Numbers (whole numbers, fractions, mixed numbers, decimals, 
and integers), 

Operations (addition, subtraction, multiplication, division, 

powers, square roots, and absolute value), 
Geometry (identification, classification, and comparison of 

geometric figures 
Algebra (algebraic variables and expressions, linear equations, 

inequalities, and graphs and functions), and 
Pr ability and Statistics (descriptive statistics, probability, 
permutations and combinations). The curriculum consists 
of 130 lessons organized into four grade levels 5 through 8. 

In the Middle School Math Curriculum, concept development 
activities provide graphic models to promote mental images of 
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concepts and insight into the relationships between concepts and 
symbolic math skills, "notice activities with consistent feedback 
reinforce the math skilh. Helps provide additional support for 
students who need U without intruding on students who do not. Drill 
activities encourage immediate recall of facts. Problem solving 
activities promote productive problem solving habits and challenge 
students to thoughtfully apply previously learned math concepts in 
real fe contexts. Rich, animated graphics demonstrate math 
concepts and principles along with prompts and questions that guide 
the student through an interactive, discovery approach to learning 
.nathematics. 

Learner Management Features 

As each student progresses through the curriculum, teachers 
can request several standard management reports which include 
information on individual student and class progress. These 
management reports indicate the lesson difficulty for each student 
and class, time on task for the student and clan, the number of trials 
each student required for mastering the activity, tte number of 
activities attempted, and number mastered for each student and 
class, and the student's relative class standing. The learner 
management system provides teachers with considerable flexibility 
in managing individual and class group courseware placement, 
assigning and sequencing activities for students and class groups, and 
in determining the frequency and the types of management reports 
desired. Management reports can be generated tor either individual 
students or class gr aps for an individual courseware activity, 
multiple activities, or the remaining activities in each student 
assignment list. The learner management capabilities also allow 
districts to correlate their own district objectives with WICAT 
courseware objectives. Teachers can also restructure the courseware 
curriculum for students and classes to better parallel the classroom 
instructional sequence. 
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Advanced Assessment and Testing Features 

The Learning Solution includes the integration of advanced, 
computerized assessment and testing capabilities for schools. WICAT 
has developed computerized predictive assessment tests based on a 
national list of assessment objectives and specific state predictive 
assessment tests for the states of Texas and Florida. Additional state 
predictive assessment tests will be developed in the future for other 
states. These predictive assessment tes's can be administered at the 
school's choice any time during the school year. The test results 
indicate v ich state assessment objectives wch student or group of 
students has mastered and which objectives are not yet mastered. 
For objectives which are not mastered, the computerized assessment 
system also provides prescriptions to appropriate textbook pages and 
to courseware lessons* These prescriptions are used by the teacher 
to more effectively help the student or group of students master the 
state mandated objectives. When students start using a courseware 
package the computerized testing capabilities can help determine 
their skills in the subject and provide appropriate placement in the 
courseware lessons. 

In the use of these computerized testing programs, WICAT has 
found the following advantages cf computerized testing over 
standard paper and pencil testing: 

• standardized administration conditions 

• ease of administration and management 

• individualized test administration 

• immediate scoring and reporting 

• non-biased scoring 

• enhanced presentation capabilities (tex t graphics, 
and audio) 

• enhanced response capabilities (multiple choice, free 
response, performance) 

• increased testing efficiency, and 

• improved test security 

WICAT has also been £ pioneer in the development of 
computerized adaptive or tailored tests of school achievement and 
ability. With a computerized adaptive test the student receives an 
initial item of average difficulty. If the student answers the item 
correctly, a more difficult item is presented. If the students answers 
the item incorrectly, a less difficult item is presented. The testing 
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process continues to adapt to the individual responses of the student. 
After each item a new estimate of the student's ability or proficiency 
in ihe subject area is estimated The testing process continues until a 
specified le r el of precision or standard error is reached and the 
testing is terminated Research results have shown that 
computerized acaptive testing can significantly reduce the amount of 
testing time and number of items required in a test by SO to 75% 
with an equal or greater level of precision of measurement. In their 
work with computerized adaptive tests, WICAT has found the 
following advantages of computerized adaptive testing over standard 
paper and pencil testing: 

• Provides more precise measurement with fewer test items 
than conventional tests 

• reduces testing time by 50% to 70% 

• tests are adapted or tailored to each individual 
student's responses 

• uses current procedures and applications of item 
response theory 

• provides equally precise measurement at all ability 
levels 

• carefully selects test items to match student ability 
levels 

• reduces frustration for low ability students and 
reduces boredom for high ability student. 

The Learning Solution also provides the schools with the 
capability to create their own computer-administered or 
computerized adaptive tests of school achievement and aptitude. 
Districts use comprehensive banks of objectives and items to select 
the specific objectives required for testing, to select the test ite:ns 
(text, graphics, and audio), to locally generate the computerized tests, 
to administer the tests and tc generate individual and group reports 
on the customized, computerized tests. Districts can also contract for 
the preparation of customized objective and item banking services 
which includes computerize banks of the district's own objectives 
and test items. 

Training and Support Features 

The Learning Solution provides on going training and support 
for each of the Learning Center installation sites. An Account 
Manager and a Technical Education Specialist work directly with each 
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installation site to meet the specific site needs for system 
information, proposals, purchr ing, installation, implementation, 
training and support. Under ne direction of the Account Manager, 
the Technical Education Specialist provides school administrator 
training, learning center manager training and teacher training 
during a three to five day period at the beginning of system 
installation. The training includes information on the Learning 
Center capabilities and features, sc eduling and implementation 
issues student behavior management, curriculum overviews, 
recommendations for implementing curricula, hands-on experience 
with the curricula being implemented, interpreting and using the 
management system reports for curriculum and testing, and 
recommendations for integrating learning centei activities with 
classroom activities. Follow-up training and continued classroom 
integration training is provided for each learning center site during a 
two-three day period a few months following system installation. 
The Account Manager and the Technical Education Specialist also 
provide on-going training and support as requested by the learning 
center site. 

Learning System Maintenance Features 

The Learning Solution includes comprehensive hardware, 
system software, courseware, and testware maintenance, WICAT 
provides a nationwide network of customer service maintenance 
technicians in addition to a 24 hour toll-free, hotline support for 
hardware, software, courseware, and testware maintenance. The 
customer service maintenance technicians provide preventive as well 
as on-going system maintenance for each learning center installation 
site, WICAT provides comprehensive, full-service maintenance 
agreements for all Learning Center sites. Learning Center managers 
are requested to call the toll-free, hotline number if any problem is 
encountered in the system hardware, system software, courseware, 
or testware When a maintenance call is received it is immediately 
dispatched to the appropriate customer service maintenance 
technician for resolution. WICAT's central maintenance dispatch 
service can provide current information on the status of any site 
maintenance request for hardware, software, courseware, or 
testware. WICAT also maintains a current staff for problem 
resolution and enhancement for any of the courseware or testwaic 
products. In addition, the Technical Education Specialists work 
directly with the learning center managers to provide training and 
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support for new system software, courseware, and test ware 
products. 

Hardware Solutions Designed for Schools 

The Learning Solution provides hardware options and solutions 
vhich are designed for schools. The Learning Center systems can 
mt . : the needs of small to moderate sized schools with the standard 
learning center configuration with 32 learning stations for a school 
with up to 350 students. The large school learning center 
configuration includes up to 64 learning stations for a school with up 
to 700 students. The smallest of the learning center configuration 
includes either 8 or 16 terminals and can meet the needs for schools 
or special education programs with up to 175 students. The modular 
configurations of the learning center system configurations provide 
for continued expansion of the learning system as the school needs 
change. The Learning Center provides schools with high fidelity 
audio, sophisticated graphics, and animation capabilities. Schools can 
also link popular microcomputers (Apples and IBM PC's and 
compatibles) as alternative learning center stations. Current 
capabilities also allow for centralized management of learning 
stations which aic located in multiple classroom locations. 

EVALUATING THE WICAT LEARNING SOLUTION 
FOR MINORITY STUDENTS 

The Learning Solution has been installed in several minority 
school districts for a period of 1-4 years. Several of these, districts 
have conducted district evaluations of the outcomes from the 
Learning Solution. Most of the available evaluations have been 
conducted using * pretest-posttest evaluation design. WICAT is 
current} developing a comprehensive evaluation plan which will 
employ experimental and control groups within six districts to 
evaluate the outcomes from implementing the Learning Solution 
(Olsen, 1987). 

The following section present* results from district -conducted 
evaluations of the Learning Solution for minority students 
additional information will be available over the next several years 
as WICAT continutes their comprehensive evaluation studies. 

ACHIEVEMENT RESULTS FROM USING THE WICAT LEARNING 
S< LUTION 
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Pharr-S*m Juan-Alamo District, TX 



The Pharr-San Juan-Alamo, TX school district has implemented 
the Learning Solution district-wide at seventeen school sites during 
the 1986-1987 school year* The Ph<urr-San Juan-Alamo Independent 
School District is located in the Rio Grande Valley of Texas. The 
student population is about 90% Hispanic. Approximately 40% of the 
school population had Limited English Proficiency at the beginning of 
the evaluation period. Each of the seventeen sites installed a 
Learning Center with 32 learning stations along with the courseware 
for Primary Reading and Reading Comprehension. Evaluations were 
conducted with between 130C to 1400 students at each grade level 3 
and 5. 

The WlCAT computerized predictive state assessment test was 
administered in October 1986 to all district third and fifth grade 
students, Individual and group reports showing mastery or non- 
mastery of state assessment objectives were provided to teachers 
Teachers also received prescriptions to courseware lessons rnd 
textbook pages fo; non-mastered objectives Teachers implemented 
the appropriate classroom and Learning Center prescriptions. The 
WICAT computerized predictive state assessment test was 
re ad ministered in January, 1987. Teachers again implemented the 
appropriate remedial prescriptions. The Texas state assessment test 
was administered in February, 1987. 

An evaluation of the Learning Solution was conducted us:ng the 
Texas Educational Assessment of Minimum Skills, the Texas state 
assessment test, admiuister?d in February, 1986 and February, 1987. 
The Febrr^jy 1986 results are prior to implementing the Learning 
Solution; the February 1987 results are after implementing the 
Learning Solution. Table 1 presents the learning outcome results 
comparing the percent of students passing the stais assessment iest 
at the district and state levels. The district wide achievement gains 

to 40% gains) are several times greater than the comparable 
state achievement gains to 8% gains). In 1986 the district results 
were either at or significantly below the state average; in 1987 the 
district results were significantly greater than the state averages. 
Table 2 presents results tor the district and state for the Limited 
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English Proficiency students. The district Limited English Proficiency 
achievement gains (23% to 47%) are significantly greater than the 
comparable state achievement gains (2% to 16%). These results show 
that the Learning Solution as very effective for districtwide 
minority students and for Limited English Proficiency minority 
students. 
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TABLE i 

EDUCATIONAL OUTCOME RESULTS 
PERFORMANCE ON THE TEXAS STATE ASSESSMENT TEST 
PHARR-SAN JUAN-ALAMO INDEPENDENT SCHOOL DISTRICT 
SPRING 1986 - SPRING 1987 

SUBJECT GRADE DISTRICT RESULTS STATE RESULTS 



READING 3 



5 



MATH 3 



WRITING 3 



55% passing 
75% passing 
20% Gain 

57% passing 
84% passing 
27% Gain 

72% passing 
90% passing 
18% Gain 

62% passing 
89% passing 
27% Gain 

49% passing 
71% passing 
22% Gain 

41% passing 
83% passing 
42% Gain 



1986) 74% passing 

1987) 79% passing 
5% Gain 

1986) 83% passing 

1987) 83% passing 
0% Gain 

1986) 80% passing 

1987) 86% passing 
6% Gain 

1986) 80% passing 

1987) 86% oassing 
6% Gain 

1986) 50% passing 

1987) 63% passing 
13% Gain 

1986) 64% passing 

1987) 68% passing 
4% Gain 
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TABLE 2 

LIMITED ENGLISH PROFICIENCY STUDENTS 
PERFORMANCE ON THE TEXAS STATE ASSESSMENT TEST 
PHaRR-SAN JUAN-ALAMO INDEPENDENT SCHOOL DISTRICT 







SPRING 1986 - 


SPRING 


1987 




SUBJECT 
READING 


GRADE 


DISTRICT RESULTS 


STATE RESULTS 




3 


29% passing 
56% passing 
27% Gain 


(1986) 
(1987) 


30% passing ( 
42% passing ( 
12% Gain 


(1986) 
(1987) 




5 


27% passing 
64% passing 
37% Gain 


(1986) 
(1987) 


40% passing I 
46% passing 
6% Gain 


(1986) 
(1987) 


MATH 


3 


59% passing 
83% passing 
34% Gain 


(1986) 
(1987) 


55% passing ( 
71% passing 
14% Gain 


(1986) 
(1987) 




5 


37% passing 
76% passing 
39% Gain 


(1986) 
(1987) 


51% passing \ 
65% passing I 
14% Gain 


(1986) 
(1987) 


WRITING 


3 


27% passing 
50% passing 
23% Gain 


(1986) 
(1987) 


26% passing ( 
40% passing ( 
14% Gain 


(1986) 
(1987) 




5 


17% passing 
64% passing 
47% Gain 


(1986) 
(1987) 


29% passing 
31% passing 
2% Gain 


(1986) 
(1987) 



252 



S3 



San Jacinto Elementary School, 

Goose Creek Independent School District, TX 



The Learning Solution was installed at the San Jacinto 
Elementary Scnool, Goose Creek Independent School District, TX, The 
San Jacinto Elementary School is a Chapter I school The school 
population is 89% minority, predominantly Hispanic. A Learning 
Center with 32 learning stations was installed along with the 
courseware for Primary Reading. The Learning Center was used by 
101 Grade 3 students for 30 minute sessions, three to four times a 
week. The WICAT predictive state assessment test was administered 
as described above for Pharr-San Juan-Alamo district. The Texas 
Educational Assessment of Minimum Skills was administered to all 
students grades 3 during February 1986 and February 1987. The 
February 1986 results are prior to implementing the Learning 
Solution. Table 3 presents the learning outcome results. These 
results show significantly greater achievement growth for the school 
(13 to 28% gains) than the comparable state results (5 to 13%). 
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TABLE 3 

EDUCATIONAL OUTCOME RESULTS 
PERFORMANCE ON THE TEXAS STATE ASSESSMENT TEST 
SAN JACINTO ELEMENTARY SCHOOL. TX 
SPRING 1986 - SPRING 1987 

SUBJECT GRADE SCHOOL RESULTS STATE RESULTS 
READING 3 61% passing (1986) 74% passing (1986) 



89% passing (1987) 
28% Gain 



79% passing (1987) 
5% 



MATH 



83% passing (1986) 
96% passing (1987) 
13% Gain 



80% passing (1986) 
86% passing (1987) 
6% Gain 



WRITING 



51% passing (1986) 
76% passing (1587) 
25% Gain 



50% passing (1986) 
63% passing (1987) 
13% Gain 
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Blackstock Junior High, Oxnard, CA 



The Learning Solution was installed in the Blackstock Junior 
High School during the 1984-1985 school year. Blackstock School is a 
6-8 grade junior high school with a total enrollment of 810 students. 
Sixty percent are minority students with the largest group being 44% 
Hispanic. The Learning Center with 30 learning stations was 
installed along with the courseware for Reading Comprehension and 
Mathematics. Students used the Learning Center for IS minute 
periods 5 times a week for reading, and 5 times a week for 
mathematics. An evaluation was conducted for 112 6th grade 
students using the Comprehensive Test of Basic Skills administered 
during the spring of 1983, 1984 and 1985, School percentile scores 
and gains were compared for Spring 1983 and Spring 1985. The 
Spring 1983 scores were prior to implementing the Learning 
Solution. Table 4 presents the learning outcome results. These 
results show learning gains of 11 to 30% over the two year period. 
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TABLE 4 

EDUCATIONAL OUTCOME RESULTS 
BLACKSTOCK SCHOOL, OXNARD, CA 
COMPREHENSIVE TEST OF BASIC SKILLS 
SPRING 1983 - SPRING 1985 



SUBJECT GRADE NATIONAL 

PERCENTILE 

READING 6 48 (1983) 

61 (1985) 
13 Percentile Gain 

MATHEMATICS 6 47 (1983) 

77 (1985) 
30 Percentile Gain 

LANGUAGE 6 49 (1983) 

ARTS 60 (1985) 

11 Percentile Gain 
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7?nos Coleman Elementary School, Chicago, IL 

The WICAT Learning Solution was installed at the Zenos 
Coleman School, Chicago, IL during the 1985-1986 school year 
Coleman school is a K-8 grade school with an enrollment of 950 
students. Nearly all of the students (98.5%) live in public housing 
projects. The entire student population is black. The school 
implemented a 32 station Learning Center including the K-8 
Mathematic and K-3 Primary Reading courseware. A total of 233 
students from grades 1-6 used the Learning Center for 90 minutes a 
week in reading and 60 minutes a week in mathematics. The Iowa 
Test of Basic Skills was administered to Grade 1 students in 
November, 1985 and April, 1986, and to Grade 2-6 students in April, 
1985 and April, 1986, Table 5 presents the learning outcome results. 
These results show grade equivalent gains twice as large with the 
learning solution compared with prior achievement gains with 
standard classroom instruction. 
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TABLE 5 
EDUCATIONAL OUTCOME RESULTS 
ZENOS COLMAN SCHOOL, CHICAGO, IL. 



Grade 1 Grade 2-6 

N=71 N=162 

MATHEMATICS LS 7,7 months gain 9,8 months gain 

in five months in nine months 

CI 3,9 months gain 4,4 months gain 

prior year prior years 

READING LS 9,1 months gain 9,5 months gain 

in five months in nine months 

Ci 6,9 months gain 4,5 months gain 

prior years prior years 

KEY 



LS = Learning Solution 
CI = Classroom Instruction 
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McCorkle Elementary School, Chicago, IL. 

The WICAT Learning Solution was installed in the McCorkle 
Elementary School, Chicago, IL. during the 1986-1987 school year. 
McCorkle Elementary School, located on Chicago's southside, is a 
preschool-8th grade elementary school with an average enrollment 
of 600 students. Approximately 90% of the students live in public 
housing projects. The student population is black. A WICAT 
Learning Center with 32 learning stations was installed along with 
the courseware for Primary Reading, Reading Comprehension, and 
Mathematics. The students used the Learning Center for five 35 
minute periods, three times a week in reading, and two times a week 
in mathematics. The Iowa Test of Basic Skills was administered to 
400 students in grades 4-8 during April, 1986 and April, 1987. 
Table 6 summarizes the learning outcome results. These results 
show grade equivalent gains which are significantly greater with the 
learning solution than with prior achievement gains with standard 
classroom instruction. 
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TABLE 6 
EDUCATIONAL OUTCOME RESULTS 
MCCORKLE ELEMENTARY SCHOOL, CHICAGO, IL. 

Gra<ie 4-8 
N=400 



MATHEMATICS LS 



8.1 months gain 
in five months 



KEY 



CI 



5.7 months gain 
prior years 



READING 



LS 



10.6 months gain 
in five months 



CI 



4.2 months gain 
prior years 



LS = Learning Solution 
CI = Classroom Instruction 
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This paper has presented a comprehensive description of the 
WICAT Learning Solution and promising achievement results for 
minority students. These achievement results were demonstrated on 
state assessment tests and standardized norm-referenced 
achievement tests. Similar significant learning gains were found 
across districts, regional locations and minority populations. 
Following are some possible explanations for these significant 
learning results. 

First, the Learning Solution includes computerized, criterion- 
referenced tests to assessing and targeting specific individual 
learning needs. These computerized tests provide prescriptions and 
placement into the courseware as well as classroom textbooks to help 
students master the instructional objectives. These computerized 
tests can be administered as often as required to provide 
assessments of student progress on specific instructional objectives. 

Second, the comprehensive courseware presented in the 
Learning Center is an effective educational supplement to the 
traditional classroom instruction. The courseware has sufficient 
scope and breadth to meet the needs of students at every grade K-12 
in mathematics reading, and language arts areas. These individual 
learning needs can be met for mainstream students, remedial 
students and gifted and talented students. The courseware also 
teaches higher order thinking and problem solving skills along with 
the basic skills instruction. The courseware provides truly 
individualized instruction with unlimited example and practice trials 
using a combination of text, graphics, high quality audio and 
animations Minority students can benefit significantly from the 
quality courseware design, step by step presentation, interactive 
example and practice opportunities, and the graphics and audio 
instructional supplements. 

Third, the Learning Solution provides teachers with a 
comprehensive instructional management system for individual 
student and class progress tracking. These management reports 
indicate student progress on test objectives and courseware 
instruction. Teachers can readily modify the student courseware 
assignment lists to better meet individual student needs. As 
teachers continue to integrate the Learning Center instruction with 
their classroom instruction the student's learning achievement will 
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improve significantly. With previous individualized instructional 
systems the teachers were often overwhelmed with the amount of 
paperwork and management time required to provide individual 
assignments, lessons and tests for each student With the 
computerized courseware the teacher can easily do the management 
work required for individualized instruction. Teachers also provide 
instructional support and individual help to students in the Learning 
Center, 

Fourth, the minority students themselves report significant 
value from using the Learning Solution. The student? report liking 
the individualized, one-on-one instruction which the courseware 
provides, the interactive practice and feedback exercises, and the 
high quality of the text, graphics and audio instruction. Students like 
to keep track of their own progress through the courseware lessons 
and testing activities. As shown above the Learning Solution also 
produces significantly student achievement and learning gains for 
minority students on state assessment tests and standardized 
achievement tests. 

This paper has described and demonstrated the benefits from 
the Learning Solution for minority students. These results offer the 
promise of significant improvement of minority student achievement 
with implementation of the Learning Solution. 
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